Welcome to WebmasterWorld Guest from 54.146.248.111

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
104.154.16.xxx
Strange access from Google IPs
8 doc_z 8:07 am Feb 24, 2015
Strange CFNetwork User-Agents
com.apple.WebKit.WebContent
4 dstiles 9:42 pm Feb 21, 2015
OrgProbe
1 keyplyr 11:46 am Feb 19, 2015
Onavo Proxy
Mobile data compressor
16 dstiles 10:44 pm Feb 17, 2015
Server Farms - August 2014
Tracking and Reporting Data Center IP Ranges [10] ( 1 2 3 4 5 6 7 8 9 10 )
285 incrediBILL 5:53 am Feb 17, 2015
WorldBrewBot
3 keyplyr 6:06 pm Feb 14, 2015
Scansafe ranges
5 dstiles 8:07 pm Feb 12, 2015
At Home with the Robots: 2015 edition
12 lucy24 8:27 am Feb 12, 2015
How can bots know the internal structure of a password protected site?
7 lamati 10:35 am Feb 7, 2015
antiquated robots
3 lucy24 10:24 pm Feb 5, 2015
acapbot/0.1
4 keyplyr 8:11 pm Feb 4, 2015
"We are not evil"
or so they say...
1 trintragula 10:53 am Feb 4, 2015
Block everything from 54. except certain bots
18 physics 8:51 pm Feb 1, 2015
Automating the server farm identification
an alternative approach[3] ( 1 2 3 )
63 trintragula 12:55 am Feb 1, 2015
Android, Frames and Google
Can android handle frames?
4 dstiles 7:33 pm Jan 31, 2015
Looks like spiders from China?
Receiving responsecode 500 errors from China visitors
6 Anders 6:49 pm Jan 29, 2015
AdvBot
9 lucy24 9:32 pm Jan 24, 2015
MegaLine, KG
1 keyplyr 9:04 am Jan 23, 2015
Odd User Agent
13 wilderness 8:32 pm Jan 22, 2015
"Mozilla/5.0 ()" - Is this used by any valid browsers?
7 physics 12:15 am Jan 19, 2015
JiveSearchBot
1 keyplyr 5:13 am Jan 14, 2015
Named distributed crawler list
6 trintragula 11:03 am Jan 9, 2015
HTTP header fields
ACCEPT and ACCEPT-CHARSET
3 dstiles 5:01 pm Jan 8, 2015
Opera Mini IP ranges
5 Xpat 5:22 pm Jan 7, 2015
Chinese exploiter out of Kansas City IP
10 slipkid 6:34 am Jan 6, 2015

Featured Threads

Hot Threads This Week

Hot Threads This Month