Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Pioneer User Agent
3 frontpage 7:28 pm July 10, 2002
  Internet Explore 5.x -
**mumble mumble** hyperactive chinese bot **mumble mumble**
6 Dreamquick 7:50 am July 10, 2002
  AltaVista NOIMAGEINDEX
name="robots" content="noimageindex" Is this new?
16 misosoph 11:24 pm July 9, 2002
  Spider Trap needs GOOD Spider list
Having setup a good spider trap I need to exclude the good ones...
15 scottspence 4:44 am July 9, 2002
  Heartbeat page requests...
.. could be a little off topic...
2 scottspence 10:04 pm July 8, 2002
  Should a spider take more than a few pages at a time?
whole site not been indexed
4 nickc001 8:10 am July 8, 2002
  DTS Agent
Anyone know what this one is?
6 jdMorgan 12:55 am July 8, 2002
  Steeler from 157.82.157.91
One of those research bots you guys hate.
7 mbauser2 10:09 am July 7, 2002
  wysiwyg
possible to ban?
12 keyplyr 8:57 am July 6, 2002
  mylinea?
2 Sinner_G 9:28 pm July 5, 2002
  SansarnRobot and S@...
Originating from 203.185.132.67
2 hanuman 8:12 am July 5, 2002
  Fast (alltheweb) - "FirstPage" - noticed this for the first time today
4 engine 11:24 pm July 4, 2002
  I need a list of Google IPs
2 tomasz 12:17 pm July 4, 2002
  Dual Proxy
what is it?
7 keyplyr 5:58 pm July 3, 2002
  Spider Bait - anyone heard of it?
apparently it hides the ? in database driven pages
3 charon 10:43 am July 3, 2002
  the NEW Hotbot/Lycos alliance
Which spiders now? No more inktomi
5 Friday 6:57 am July 3, 2002
  Lachesis
anyone heard of it?
5 incywincy 10:45 pm July 2, 2002
  Inktomi bot?
3 Weblamer 8:56 pm July 2, 2002
  Pompos/1.0 pompos@iliad.fr
Anybody seen this?
5 volatilegx 9:55 pm July 1, 2002
  Who's Libby
and why does she want to be my friend?
3 idiotgirl 8:09 pm July 1, 2002
  Is this valid robots.txt?
valid robots.txt * asterisk disallow
3 pwc21 3:36 pm July 1, 2002
  display error pages to banned users
.htaccess too aggressive
6 idiotgirl 7:04 pm June 30, 2002
  Spider? choking on javascript
Huge increase in 404's
2 Robert_Scott 10:20 am June 29, 2002
  Possible Spambot wih user agent "MFC Library"
2 Bluestreak 10:39 pm June 27, 2002
  Annoying and unusual bot activity from other countries
3 Bluestreak 2:43 pm June 27, 2002