Welcome to WebmasterWorld Guest from 54.197.75.83

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Exabot-Thumbnails redux
3 lucy24 9:29 pm Mar 13, 2015
Blocking all traffic from amazonaws.com
I want to block all traffic
15 Kratos 1:14 pm Mar 11, 2015
BingAdsAdInsightBot
New msnbot Spawn
7 Pfui 3:18 am Mar 11, 2015
iParadigms
TurnItIn
8 keyplyr 4:09 pm Mar 9, 2015
kb58.cn
Is there a bot to this
9 Marshall 1:37 pm Mar 7, 2015
Newbie: Need help with server farms and IP ranges
25 MrSavage 2:11 am Mar 6, 2015
PageFetcher-Google-CoOp
14 aristotle 2:04 am Mar 6, 2015
Gulper Web Bot
stonybrook.edu
8 keyplyr 4:43 am Mar 3, 2015
This thing shows up every 30-40 minutes on two of my sites
16 aristotle 2:09 am Mar 3, 2015
Image Bot Names?
4 keyplyr 10:46 pm Feb 27, 2015
Apple search engine bot?
24 johnhh 8:49 pm Feb 25, 2015
IANA Private CIDRs
Respectable Spooks only?
4 Angonasec 7:14 pm Feb 25, 2015
OrgProbe Bot
Warning
12 Angonasec 2:33 pm Feb 25, 2015
104.154.16.xxx
Strange access from Google IPs
8 doc_z 8:07 am Feb 24, 2015
Strange CFNetwork User-Agents
com.apple.WebKit.WebContent
4 dstiles 9:42 pm Feb 21, 2015
OrgProbe
1 keyplyr 11:46 am Feb 19, 2015
Onavo Proxy
Mobile data compressor
16 dstiles 10:44 pm Feb 17, 2015
Server Farms - August 2014
Tracking and Reporting Data Center IP Ranges [10] ( 1 2 3 4 5 6 7 8 9 10 )
285 incrediBILL 5:53 am Feb 17, 2015
WorldBrewBot
3 keyplyr 6:06 pm Feb 14, 2015
Scansafe ranges
5 dstiles 8:07 pm Feb 12, 2015
At Home with the Robots: 2015 edition
12 lucy24 8:27 am Feb 12, 2015
How can bots know the internal structure of a password protected site?
7 lamati 10:35 am Feb 7, 2015
antiquated robots
3 lucy24 10:24 pm Feb 5, 2015
acapbot/0.1
4 keyplyr 8:11 pm Feb 4, 2015
"We are not evil"
or so they say...
1 trintragula 10:53 am Feb 4, 2015