Welcome to WebmasterWorld Guest from 54.226.146.15

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
bad spider, no UA, no read of robots.txt
where's my newspaper....
6 dcheney 5:55 am Jul 11, 2002
bigip1a-snat.sv.av.com
who is from altavista ?
2 berno 7:32 pm Jul 10, 2002
Pioneer User Agent
3 frontpage 7:28 pm Jul 10, 2002
Internet Explore 5.x -
**mumble mumble** hyperactive chinese bot **mumble mumble**
6 Dreamquick 7:50 am Jul 10, 2002
AltaVista NOIMAGEINDEX
name="robots" content="noimageindex" Is this new?
16 misosoph 11:24 pm Jul 9, 2002
Spider Trap needs GOOD Spider list
Having setup a good spider trap I need to exclude the good ones...
15 scottspence 4:44 am Jul 9, 2002
Heartbeat page requests...
.. could be a little off topic...
2 scottspence 10:04 pm Jul 8, 2002
Should a spider take more than a few pages at a time?
whole site not been indexed
4 nickc001 8:10 am Jul 8, 2002
DTS Agent
Anyone know what this one is?
6 jdMorgan 12:55 am Jul 8, 2002
Steeler from 157.82.157.91
One of those research bots you guys hate.
7 mbauser2 10:09 am Jul 7, 2002
wysiwyg
possible to ban?
12 keyplyr 8:57 am Jul 6, 2002
mylinea?
2 Sinner_G 9:28 pm Jul 5, 2002
SansarnRobot and S@...
Originating from 203.185.132.67
2 hanuman 8:12 am Jul 5, 2002
Fast (alltheweb) - "FirstPage" - noticed this for the first time today
4 engine 11:24 pm Jul 4, 2002
I need a list of Google IPs
2 tomasz 12:17 pm Jul 4, 2002
Dual Proxy
what is it?
7 keyplyr 5:58 pm Jul 3, 2002
Spider Bait - anyone heard of it?
apparently it hides the ? in database driven pages
3 charon 10:43 am Jul 3, 2002
the NEW Hotbot/Lycos alliance
Which spiders now? No more inktomi
5 Friday 6:57 am Jul 3, 2002
Lachesis
anyone heard of it?
5 incywincy 10:45 pm Jul 2, 2002
Inktomi bot?
3 Weblamer 8:56 pm Jul 2, 2002
Pompos/1.0 pompos@iliad.fr
Anybody seen this?
5 volatilegx 9:55 pm Jul 1, 2002
Who's Libby
and why does she want to be my friend?
3 idiotgirl 8:09 pm Jul 1, 2002
Is this valid robots.txt?
valid robots.txt * asterisk disallow
3 pwc21 3:36 pm Jul 1, 2002
display error pages to banned users
.htaccess too aggressive
6 idiotgirl 7:04 pm Jun 30, 2002
Spider? choking on javascript
Huge increase in 404's
2 Robert_Scott 10:20 am Jun 29, 2002

Featured Threads

Hot Threads This Week

Hot Threads This Month