Welcome to WebmasterWorld Guest from 54.158.166.6

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Atspider
Spam spider gone mad
2 estaquieto 2:57 pm May 28, 2002
New Spider?
Gigabot/1.0
2 dcheney 9:15 am May 25, 2002
Win32; WinHttp.WinHttpRequest.5
67.113.225.66
4 wilderness 12:39 am May 25, 2002
Block and Complain success
Sometimes they do clean up their act
5 jdMorgan 5:31 am May 24, 2002
Interesting spiders from AV & Mercator
17 Air 3:47 pm May 23, 2002
Cruizer IV
Does anyone know it
3 olwen 8:38 am May 22, 2002
What is this user agent?
lwp-request/2.01
3 korkus2000 9:15 pm May 21, 2002
Zealbot
LookSmart Spider
2 Key_Master 6:43 am May 21, 2002
How do I disallow spiders?
I want to know what code to put on a page to not allow spiders to crawl.
2 Mitch 5:03 pm May 20, 2002
another one to deny
7 littleman 5:31 am May 20, 2002
Tcl http client package 1.0
What is this?
4 misosoph 11:22 am May 19, 2002
Anyone Seen This INK bot
3 dstanovic 11:01 am May 19, 2002
Ask Jeeves spider?
2 korkus2000 1:30 pm May 18, 2002
Anyone seen this Ask Jeeves spider?
Ask Jeeves spider - why?
6 IanTurner 1:55 am May 18, 2002
Coding a spider
How to give yourself an agent name
6 chris_f 6:53 pm May 17, 2002
Inktomi
New C class or did I forget to add this one again?
2 msgraph 4:47 pm May 17, 2002
webcollage variation
collage.cgi/1.82
4 misosoph 2:09 pm May 17, 2002
Unknown Crawler
193.95.83.130
9 chris_f 8:07 am May 15, 2002
spider-loh-tc063.proxy.aol.com
What Is This
5 Funfone 8:44 pm May 14, 2002
AltaVista Spider?
trek17.sv.av.com
3 HyperGeek 4:27 pm May 14, 2002
Unknown - Lots of activity
66.28.250.176
3 HyperGeek 4:02 pm May 14, 2002
Two mystery spiders
Heavy crawlers
15 Everyman 5:09 am May 14, 2002
Need help ID ing a spider
2 RobertS 12:35 pm May 13, 2002
What is Dsurf?
6 Brett_Tabke 9:41 am May 12, 2002
Gozilla/4.0
Just looking for formmail.pl
7 bobriggs 9:30 am May 12, 2002