Welcome to WebmasterWorld Guest from 54.167.153.63

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Windows NT 9.0: Back To The Future?
Guess I missed something while I was out...
1 iamzippy 3:51 pm Mar 17, 2012
SWEbot
8 tangor 10:24 pm Mar 16, 2012
Wikimpress - dumbest scraper project ever
5 DeeCee 9:32 am Mar 16, 2012
Yahoo! Slurp
Two versions; one ignores robots.txt[2] ( 1 2 )
50 Pfui 10:28 pm Mar 15, 2012
Everhost
Mozilla is out there, somewhere...
1 iamzippy 8:55 am Mar 13, 2012
AOL UA's
3 wilderness 2:38 am Mar 13, 2012
very unusual swarm of crawlers
14 wilderness 1:59 am Mar 13, 2012
Who.is Bot
13 iamzippy 4:19 pm Mar 12, 2012
NET Block assignments
4 wilderness 10:27 pm Mar 11, 2012
Hostnoc
3 wilderness 8:05 pm Mar 11, 2012
getting hammered by tide544.microsoft.com today
5 bakedjake 10:34 pm Mar 10, 2012
At Home with the Robots
25 lucy24 6:15 pm Mar 10, 2012
of pipl and robots
18 lucy24 7:52 am Mar 10, 2012
Is this spider a spam bot?
12 Mario155 2:59 pm Mar 9, 2012
YahooCacheSystem - what is it?
29 motorhaven 10:24 pm Mar 8, 2012
Googlebot trying port 21
4 Peter 3:16 pm Mar 8, 2012
WeSEE:Search
7 keyplyr 11:43 pm Mar 6, 2012
BingWeb
7 Pfui 10:49 pm Mar 2, 2012
EdisterBot
22 keyplyr 11:47 pm Feb 29, 2012
url(data:image/
G-Strings Revisited
15 Pfui 8:05 pm Feb 29, 2012
Strange user agent
18 KevinV 2:14 pm Feb 29, 2012
setup akl.exe
3 wilderness 5:22 am Feb 29, 2012
HTTrack / Virgin Media
[2] ( 1 2 )
31 cyberdyne 10:10 am Feb 27, 2012
MireoBot
1 keyplyr 12:14 am Feb 27, 2012
No more free ranges
IANA has run out of ivp4 ranges
6 dstiles 10:22 pm Feb 24, 2012

Featured Threads

Hot Threads This Week

Hot Threads This Month