Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
htaccess regex syntax
7 mattie 7:12 pm Dec 15, 2004
StanleyWebSpider/1.0
4 Ankheg 12:34 am Dec 15, 2004
Netcraft Web Server Survey
What's this one?
9 Rollo 5:28 am Dec 12, 2004
MAXOMObot Video Search rides the 'Net.
( Another Nutch. )
4 pendanticist 2:50 am Dec 11, 2004
sna-0.0.1
9 wilderness 2:44 am Dec 11, 2004
Mozilla/5.0 (Sonar) Java
100k hits yesterday
4 petez1 11:19 pm Dec 10, 2004
Slurp
Yahoo is spidering on new Class Cs
2 volatilegx 8:19 pm Dec 10, 2004

7 Strange 3:00 pm Dec 10, 2004
William Lu
4 wilderness 1:19 pm Dec 10, 2004
UA's with email addresses
5 JAB_Creations 12:17 am Dec 8, 2004
StackRambler/2.0
From then, 'till now.
3 pendanticist 9:51 pm Dec 7, 2004
Mozilla/3.01 (compatible;)
4 JAB_Creations 7:39 pm Dec 7, 2004
Ocelli/1.1
One aggressive spider!
6 jam13 9:35 am Dec 6, 2004
Requests for the same 23 pages
not sure if it is a 'bot or what...
9 carfac 3:47 am Dec 6, 2004
Looking for data on internet filters
12 volatilegx 8:07 pm Dec 4, 2004
Hotbot-Lycos Robot
4 jdancing 2:49 pm Dec 4, 2004
"GoogleBot Cloak"
Robot showed up in my tracking, "GoogleBot Cloak", what is it?
4 aghill 12:21 pm Dec 3, 2004
Ripley's
"Believe it or not"
2 wilderness 3:44 am Dec 3, 2004
HTtrack and Webpix
Both his my site, rapid fire.
5 larryhatch 10:21 pm Nov 29, 2004
pcnbot?
5 sqlgod 9:18 pm Nov 29, 2004
SearchByUsa/2
Robots.txt -yes.
4 pendanticist 2:10 am Nov 29, 2004
gazz/5.0
Japan
3 Staffa 11:49 am Nov 28, 2004
CreativeCommons
3 wilderness 3:50 am Nov 28, 2004
User-Agent: OmniWeb
Disallowed
4 guitaristinus 12:20 pm Nov 27, 2004
mozilla/4.0 (compatible; cerberian drtrs version-3.1-build-16)
Again!
6 DoppyNL 10:27 pm Nov 26, 2004