Welcome to WebmasterWorld Guest from 23.20.221.93

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
V5 HTTP/S Component
Who is that?
3 frontpage 3:57 pm Sep 22, 2002
Spider from multiple IP's all at once
7 carfac 2:07 am Sep 22, 2002
63.215.240.27 (unknown.level3.net)
Scooter/2.0
6 Key_Master 3:45 pm Sep 20, 2002
av
4 PeteU 7:21 pm Sep 19, 2002
128.242.197.101
I am paranoid but . . .
4 rogerRabbit 10:54 am Sep 19, 2002
MSIE 5.01 requesting robots.txt?
anyone who knows this?
3 bull 3:37 pm Sep 18, 2002
searching for formmail
another candidate to ban?
2 bull 9:43 am Sep 18, 2002
The number of googles spiders
The number of googles spiders what does it stands for
2 Ove 7:14 am Sep 18, 2002
minibot
any ideas?
2 Finder 11:10 pm Sep 17, 2002
Mysterious Spider
5 tesla 7:44 pm Sep 17, 2002
Ask Jeeves
Ask Jeeves disregarding robots.txt
16 Scooter24 5:00 pm Sep 17, 2002
Ill-behaved libwww-perl UA
Reads and ignores robots.txt
3 jdMorgan 4:15 pm Sep 17, 2002
Scooter ignoring one page entry in robots
how can i get it to not index this page?
8 semick 7:48 pm Sep 16, 2002
Is this a user... or a bunch of people?
2 carfac 6:57 pm Sep 14, 2002
MFC Foundation Class Library 4.0
3 Scooter24 2:21 pm Sep 14, 2002
Polybot
Old News?
3 brotherhood_of_LAN 11:14 pm Sep 12, 2002
Where are these coming from?
VERY bad spiders attacking my site...
9 irock 11:11 pm Sep 12, 2002
New Marketing Campaign Bot
won't go away...
2 fiestagirl 6:17 am Sep 12, 2002
Inktomi poking around
209.131.51.166
4 fiestagirl 11:42 pm Sep 10, 2002
ec009.inktomi.com
7 littleman 7:56 pm Sep 10, 2002
SearchTone
1 Scooter24 10:37 am Sep 10, 2002
Is there a list of user-agents from robots
9 martin 8:20 am Sep 10, 2002
dloader(NaverRobot)/1.0
Anyone know what this is?
4 Finder 10:23 pm Sep 9, 2002
NaverRobot/1.0 Seems Polite but Weird
3 carfac 10:20 pm Sep 9, 2002
Advice about Visitors with No UA
If they don't tell you who they are...
12 carfac 9:20 pm Sep 9, 2002