Welcome to WebmasterWorld Guest from 54.221.30.139

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Image Bot Names?
4 keyplyr 10:46 pm Feb 27, 2015
Apple search engine bot?
24 johnhh 8:49 pm Feb 25, 2015
IANA Private CIDRs
Respectable Spooks only?
4 Angonasec 7:14 pm Feb 25, 2015
OrgProbe Bot
Warning
12 Angonasec 2:33 pm Feb 25, 2015
104.154.16.xxx
Strange access from Google IPs
8 doc_z 8:07 am Feb 24, 2015
Strange CFNetwork User-Agents
com.apple.WebKit.WebContent
4 dstiles 9:42 pm Feb 21, 2015
OrgProbe
1 keyplyr 11:46 am Feb 19, 2015
Onavo Proxy
Mobile data compressor
16 dstiles 10:44 pm Feb 17, 2015
Server Farms - August 2014
Tracking and Reporting Data Center IP Ranges [10] ( 1 2 3 4 5 6 7 8 9 10 )
285 incrediBILL 5:53 am Feb 17, 2015
WorldBrewBot
3 keyplyr 6:06 pm Feb 14, 2015
Scansafe ranges
5 dstiles 8:07 pm Feb 12, 2015
At Home with the Robots: 2015 edition
12 lucy24 8:27 am Feb 12, 2015
How can bots know the internal structure of a password protected site?
7 lamati 10:35 am Feb 7, 2015
antiquated robots
3 lucy24 10:24 pm Feb 5, 2015
acapbot/0.1
4 keyplyr 8:11 pm Feb 4, 2015
"We are not evil"
or so they say...
1 trintragula 10:53 am Feb 4, 2015
Block everything from 54. except certain bots
18 physics 8:51 pm Feb 1, 2015
Automating the server farm identification
an alternative approach[3] ( 1 2 3 )
63 trintragula 12:55 am Feb 1, 2015
Android, Frames and Google
Can android handle frames?
4 dstiles 7:33 pm Jan 31, 2015
Looks like spiders from China?
Receiving responsecode 500 errors from China visitors
6 Anders 6:49 pm Jan 29, 2015
AdvBot
9 lucy24 9:32 pm Jan 24, 2015
MegaLine, KG
1 keyplyr 9:04 am Jan 23, 2015
Odd User Agent
13 wilderness 8:32 pm Jan 22, 2015
"Mozilla/5.0 ()" - Is this used by any valid browsers?
7 physics 12:15 am Jan 19, 2015
JiveSearchBot
1 keyplyr 5:13 am Jan 14, 2015

Featured Threads

Hot Threads This Week

Hot Threads This Month