Welcome to WebmasterWorld Guest from 54.242.63.214

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
64.140.48.30
Who should this be
6 BrunoDillen 5:28 pm Jul 16, 2002
Fetch API Request - Help!
I know more about it but I'm even more puzzled now!
9 Dreamquick 3:43 pm Jul 15, 2002
What is Indy library and can I get them to go away
12 korkus2000 2:37 pm Jul 14, 2002
New Wisenut Spider
7 fiestagirl 6:10 pm Jul 13, 2002
larbin (samualt9@bigfoot.com)
Anyone seen this before
5 fathom 6:17 am Jul 13, 2002
66.196.72.58
2 Visit_Thailand 1:57 am Jul 13, 2002
Inktomi Hosts
Any info on Inktomi identification?
2 cnegs44 9:06 am Jul 12, 2002
ipd/1.0
Has anyone else seen this one?
10 Pushycat 2:01 pm Jul 11, 2002
bad spider, no UA, no read of robots.txt
where's my newspaper....
6 dcheney 5:55 am Jul 11, 2002
bigip1a-snat.sv.av.com
who is from altavista ?
2 berno 7:32 pm Jul 10, 2002
Pioneer User Agent
3 frontpage 7:28 pm Jul 10, 2002
Internet Explore 5.x -
**mumble mumble** hyperactive chinese bot **mumble mumble**
6 Dreamquick 7:50 am Jul 10, 2002
AltaVista NOIMAGEINDEX
name="robots" content="noimageindex" Is this new?
16 misosoph 11:24 pm Jul 9, 2002
Spider Trap needs GOOD Spider list
Having setup a good spider trap I need to exclude the good ones...
15 scottspence 4:44 am Jul 9, 2002
Heartbeat page requests...
.. could be a little off topic...
2 scottspence 10:04 pm Jul 8, 2002
Should a spider take more than a few pages at a time?
whole site not been indexed
4 nickc001 8:10 am Jul 8, 2002
DTS Agent
Anyone know what this one is?
6 jdMorgan 12:55 am Jul 8, 2002
Steeler from 157.82.157.91
One of those research bots you guys hate.
7 mbauser2 10:09 am Jul 7, 2002
wysiwyg
possible to ban?
12 keyplyr 8:57 am Jul 6, 2002
mylinea?
2 Sinner_G 9:28 pm Jul 5, 2002
SansarnRobot and S@...
Originating from 203.185.132.67
2 hanuman 8:12 am Jul 5, 2002
Fast (alltheweb) - "FirstPage" - noticed this for the first time today
4 engine 11:24 pm Jul 4, 2002
I need a list of Google IPs
2 tomasz 12:17 pm Jul 4, 2002
Dual Proxy
what is it?
7 keyplyr 5:58 pm Jul 3, 2002
Spider Bait - anyone heard of it?
apparently it hides the ? in database driven pages
3 charon 10:43 am Jul 3, 2002

Featured Threads

Hot Threads This Week

Hot Threads This Month