Welcome to WebmasterWorld Guest from 23.22.206.103

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
www.go2-yellowpages.com
Deep crawling for awhile now, and incorrectly
4 savvy1 8:11 pm Sep 27, 2002
Spiders/Bots and Javascript
Spiders without javascript indexing our "javascript required" page
6 Advantex 8:46 pm Sep 25, 2002
Caught another "Trademark Search" robot
3 carfac 6:44 pm Sep 25, 2002
antibot-V1.1.5/i586-linux-2.2
belongs to AntiSearch
8 volatilegx 5:32 pm Sep 25, 2002
NetMechanic V3.0
Seems to be spidering?
2 Nick_W 8:56 am Sep 24, 2002
I am getting hit really hard...
ia_archiver
8 mack 4:27 pm Sep 23, 2002
Wired2000
any infos, ideas?
1 bull 4:01 pm Sep 23, 2002
sherlock/1.3 httpget/1.3
2 frontpage 6:57 pm Sep 22, 2002
Scrubby 2.2
Is this a new spider?
2 Nick_W 4:37 pm Sep 22, 2002
V5 HTTP/S Component
Who is that?
3 frontpage 3:57 pm Sep 22, 2002
Spider from multiple IP's all at once
7 carfac 2:07 am Sep 22, 2002
63.215.240.27 (unknown.level3.net)
Scooter/2.0
6 Key_Master 3:45 pm Sep 20, 2002
av
4 PeteU 7:21 pm Sep 19, 2002
128.242.197.101
I am paranoid but . . .
4 rogerRabbit 10:54 am Sep 19, 2002
MSIE 5.01 requesting robots.txt?
anyone who knows this?
3 bull 3:37 pm Sep 18, 2002
searching for formmail
another candidate to ban?
2 bull 9:43 am Sep 18, 2002
The number of googles spiders
The number of googles spiders what does it stands for
2 Ove 7:14 am Sep 18, 2002
minibot
any ideas?
2 Finder 11:10 pm Sep 17, 2002
Mysterious Spider
5 tesla 7:44 pm Sep 17, 2002
Ask Jeeves
Ask Jeeves disregarding robots.txt
16 Scooter24 5:00 pm Sep 17, 2002
Ill-behaved libwww-perl UA
Reads and ignores robots.txt
3 jdMorgan 4:15 pm Sep 17, 2002
Scooter ignoring one page entry in robots
how can i get it to not index this page?
8 semick 7:48 pm Sep 16, 2002
Is this a user... or a bunch of people?
2 carfac 6:57 pm Sep 14, 2002
MFC Foundation Class Library 4.0
3 Scooter24 2:21 pm Sep 14, 2002
Polybot
Old News?
3 brotherhood_of_LAN 11:14 pm Sep 12, 2002