Welcome to WebmasterWorld Guest from 54.162.239.134

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
antiquated robots
3 lucy24 10:24 pm Feb 5, 2015
acapbot/0.1
4 keyplyr 8:11 pm Feb 4, 2015
"We are not evil"
or so they say...
1 trintragula 10:53 am Feb 4, 2015
Block everything from 54. except certain bots
18 physics 8:51 pm Feb 1, 2015
Automating the server farm identification
an alternative approach[3] ( 1 2 3 )
63 trintragula 12:55 am Feb 1, 2015
Android, Frames and Google
Can android handle frames?
4 dstiles 7:33 pm Jan 31, 2015
Looks like spiders from China?
Receiving responsecode 500 errors from China visitors
6 Anders 6:49 pm Jan 29, 2015
AdvBot
9 lucy24 9:32 pm Jan 24, 2015
MegaLine, KG
1 keyplyr 9:04 am Jan 23, 2015
Odd User Agent
13 wilderness 8:32 pm Jan 22, 2015
"Mozilla/5.0 ()" - Is this used by any valid browsers?
7 physics 12:15 am Jan 19, 2015
JiveSearchBot
1 keyplyr 5:13 am Jan 14, 2015
Named distributed crawler list
6 trintragula 11:03 am Jan 9, 2015
HTTP header fields
ACCEPT and ACCEPT-CHARSET
3 dstiles 5:01 pm Jan 8, 2015
Opera Mini IP ranges
5 Xpat 5:22 pm Jan 7, 2015
Chinese exploiter out of Kansas City IP
10 slipkid 6:34 am Jan 6, 2015
Bad Bot Blackhole - Blocking Ranges
14 SerpsGuy 1:21 am Jan 1, 2015
Baidu UA
5 wilderness 10:05 am Dec 31, 2014
VisiStat
Misbehaving San Jose, CA Brat blocking itself
3 Hobbs 10:01 am Dec 31, 2014
Google-Fake Directories testing 404
23 wilderness 5:10 am Dec 18, 2014
another one for the profilers
[2] ( 1 2 )
38 lucy24 3:30 am Dec 14, 2014
Ddos attack - what to do
21 doc_z 12:37 pm Dec 13, 2014
Why so many China bots?
13 roshaoar 11:57 pm Dec 8, 2014
NerdyBot and the Google Cloud
Google range with strange referrer[2] ( 1 2 )
54 dupres01 11:40 am Dec 5, 2014
PHP Vulnerability
3 wilderness 10:09 am Nov 28, 2014

Featured Threads

Hot Threads This Week

Hot Threads This Month