Welcome to WebmasterWorld Guest from 54.167.153.63

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
ldspider
code.google.com
3 Pfui 11:53 pm Jan 27, 2012
WBSearchBot
8 Pfui 10:45 pm Jan 27, 2012
HTTwaddle
6 lucy24 11:22 am Jan 25, 2012
ucsb seclab crawler
8 Pfui 9:02 pm Jan 22, 2012
Vancouver 0.9
3 MxAngel 10:45 am Jan 20, 2012
AWS look out.
Not a spider report, a warning of increased activity!
1 tangor 6:26 am Jan 19, 2012
SpiderLing
2 Pfui 6:58 am Jan 18, 2012
Gigabot Revisited
4 Pfui 5:53 am Jan 18, 2012
utopic
1 Pfui 7:07 pm Jan 17, 2012
obot
IBM iss.net IPs
5 Pfui 4:55 am Jan 16, 2012
EuripBot
3 keyplyr 8:46 pm Jan 14, 2012
MS Crawler Hiding as a Browser
14 incrediBILL 4:19 am Jan 11, 2012
Is this a valid user agent? and how do I block it?
Is this a valid user agent? and how do I block it?
12 spiritualseo 4:59 am Jan 10, 2012
Nook
2 Pfui 7:22 pm Jan 8, 2012
Servers using open proxies
4 dstiles 3:21 pm Jan 7, 2012
boia.org
tools request, ignore robots.txt
2 Pfui 7:06 am Jan 7, 2012
Googlebot-richsnippets
2 Pfui 2:37 am Jan 7, 2012
MFE expand
3 Pfui 12:44 am Jan 7, 2012
Vocus
Hocus Pocus
2 Pfui 11:17 pm Jan 6, 2012
aiHitBot
1 Pfui 8:46 pm Jan 6, 2012
Quora Link Preview
1 Pfui 7:30 pm Jan 6, 2012
Kindle & Amazon IPs?
28 keyplyr 8:45 pm Jan 5, 2012
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
9 Pfui 8:41 pm Jan 5, 2012
Automatically detect crawlers / bots
13 Globetrotter 3:03 pm Jan 5, 2012
Semrushbot/0.9
5 tangor 11:12 pm Jan 4, 2012

Featured Threads

Hot Threads This Week

Hot Threads This Month