Welcome to WebmasterWorld Guest from 54.226.32.234

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Using mod_rewrite to stop bad spiders
Where is it?
5 transistor 12:47 am May 30, 2002
What is too much...
2 opulyst 11:55 pm May 29, 2002
Legit Spider Usage?
Are there ways I can use spiders that don't cause problems for site owners?
2 OpeRabbit 8:40 pm May 29, 2002
Identifying These IP's
Totally Skewed my Log Report
16 JanCounselor 6:23 pm May 29, 2002
Atspider
Spam spider gone mad
2 estaquieto 2:57 pm May 28, 2002
New Spider?
Gigabot/1.0
2 dcheney 9:15 am May 25, 2002
Win32; WinHttp.WinHttpRequest.5
67.113.225.66
4 wilderness 12:39 am May 25, 2002
Block and Complain success
Sometimes they do clean up their act
5 jdMorgan 5:31 am May 24, 2002
Interesting spiders from AV & Mercator
17 Air 3:47 pm May 23, 2002
Cruizer IV
Does anyone know it
3 olwen 8:38 am May 22, 2002
What is this user agent?
lwp-request/2.01
3 korkus2000 9:15 pm May 21, 2002
Zealbot
LookSmart Spider
2 Key_Master 6:43 am May 21, 2002
How do I disallow spiders?
I want to know what code to put on a page to not allow spiders to crawl.
2 Mitch 5:03 pm May 20, 2002
another one to deny
7 littleman 5:31 am May 20, 2002
Tcl http client package 1.0
What is this?
4 misosoph 11:22 am May 19, 2002
Anyone Seen This INK bot
3 dstanovic 11:01 am May 19, 2002
Ask Jeeves spider?
2 korkus2000 1:30 pm May 18, 2002
Anyone seen this Ask Jeeves spider?
Ask Jeeves spider - why?
6 IanTurner 1:55 am May 18, 2002
Coding a spider
How to give yourself an agent name
6 chris_f 6:53 pm May 17, 2002
Inktomi
New C class or did I forget to add this one again?
2 msgraph 4:47 pm May 17, 2002
webcollage variation
collage.cgi/1.82
4 misosoph 2:09 pm May 17, 2002
Unknown Crawler
193.95.83.130
9 chris_f 8:07 am May 15, 2002
spider-loh-tc063.proxy.aol.com
What Is This
5 Funfone 8:44 pm May 14, 2002
AltaVista Spider?
trek17.sv.av.com
3 HyperGeek 4:27 pm May 14, 2002
Unknown - Lots of activity
66.28.250.176
3 HyperGeek 4:02 pm May 14, 2002

Featured Threads

Hot Threads This Week

Hot Threads This Month