Welcome to WebmasterWorld Guest from 23.23.53.177

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
AltaVista NOIMAGEINDEX
name="robots" content="noimageindex" Is this new?
16 misosoph 11:24 pm Jul 9, 2002
Spider Trap needs GOOD Spider list
Having setup a good spider trap I need to exclude the good ones...
15 scottspence 4:44 am Jul 9, 2002
Heartbeat page requests...
.. could be a little off topic...
2 scottspence 10:04 pm Jul 8, 2002
Should a spider take more than a few pages at a time?
whole site not been indexed
4 nickc001 8:10 am Jul 8, 2002
DTS Agent
Anyone know what this one is?
6 jdMorgan 12:55 am Jul 8, 2002
Steeler from 157.82.157.91
One of those research bots you guys hate.
7 mbauser2 10:09 am Jul 7, 2002
wysiwyg
possible to ban?
12 keyplyr 8:57 am Jul 6, 2002
mylinea?
2 Sinner_G 9:28 pm Jul 5, 2002
SansarnRobot and S@...
Originating from 203.185.132.67
2 hanuman 8:12 am Jul 5, 2002
Fast (alltheweb) - "FirstPage" - noticed this for the first time today
4 engine 11:24 pm Jul 4, 2002
I need a list of Google IPs
2 tomasz 12:17 pm Jul 4, 2002
Dual Proxy
what is it?
7 keyplyr 5:58 pm Jul 3, 2002
Spider Bait - anyone heard of it?
apparently it hides the ? in database driven pages
3 charon 10:43 am Jul 3, 2002
the NEW Hotbot/Lycos alliance
Which spiders now? No more inktomi
5 Friday 6:57 am Jul 3, 2002
Lachesis
anyone heard of it?
5 incywincy 10:45 pm Jul 2, 2002
Inktomi bot?
3 Weblamer 8:56 pm Jul 2, 2002
Pompos/1.0 pompos@iliad.fr
Anybody seen this?
5 volatilegx 9:55 pm Jul 1, 2002
Who's Libby
and why does she want to be my friend?
3 idiotgirl 8:09 pm Jul 1, 2002
Is this valid robots.txt?
valid robots.txt * asterisk disallow
3 pwc21 3:36 pm Jul 1, 2002
display error pages to banned users
.htaccess too aggressive
6 idiotgirl 7:04 pm Jun 30, 2002
Spider? choking on javascript
Huge increase in 404's
2 Robert_Scott 10:20 am Jun 29, 2002
Possible Spambot wih user agent "MFC Library"
2 Bluestreak 10:39 pm Jun 27, 2002
Annoying and unusual bot activity from other countries
3 Bluestreak 2:43 pm Jun 27, 2002
List of recent spiders
Unknown to me
10 Thierry_Zoller 9:24 pm Jun 26, 2002
VB OpenUrl
Anyone seen this one?
3 jdMorgan 8:04 pm Jun 25, 2002