Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Whois.sc bot goes undercover
No way to opt out?
6 fiestagirl 9:40 pm Feb 20, 2003
  Follow Up ti Incy Wincy
Now called "NetResearchServer"
4 carfac 12:46 am Feb 20, 2003
  Any info on an Ip address
2 c0n0r 2:44 am Feb 19, 2003
  Spiders
Which page is search first.
2 fashezee 1:18 am Feb 19, 2003
  Identify Robots vs Humans?
How to get an accurate visitor count?
6 peterdaly 11:19 pm Feb 18, 2003
  How to Defeat Bad Web Robots With Apache
very large site about fighting bad robots
2 thermoman 7:22 pm Feb 17, 2003
  which SE they belong to?
2 taj79 12:23 pm Feb 17, 2003
  A REAL B$#@$%^#%&
Viral Marketing
12 carfac 11:00 pm Feb 16, 2003
  Bot Database
8 digitaleus 2:34 pm Feb 16, 2003
  63.148.99.247
[2] ( 1 2 )
39 palmpal 6:45 am Feb 15, 2003
  Anyone knows this bot: "Mozilla/4.0 efp@gmx.net"?
question about new bot efp@gmx.net
14 aleksl 12:10 am Feb 14, 2003
  Domanova
2 wilderness 11:47 am Feb 13, 2003
  Arin Whois
net searches
4 wilderness 4:14 am Feb 12, 2003
  Slow Spy Bot
Not sure from Where... or why...
5 carfac 3:53 am Feb 12, 2003
  Who is this?
Took a lot of bandwidth
4 spikedo55 7:30 pm Feb 11, 2003
  Dart+Communications+PowerTCP
7 paul_london 12:06 pm Feb 11, 2003
  why would a satellite proxy/cache trip spider trap
can't figure out if this was a harmless user or bad bot
3 amznVibe 10:17 pm Feb 10, 2003
  12.148.209.196
Who is this rude character?
10 Powdork 10:05 pm Feb 10, 2003
  EasyWebPromotion bot
How would I block this spider?
3 robjones 2:54 pm Feb 9, 2003
  anyone familiar with CHTML
chtml
3 coolcreep 8:12 am Feb 9, 2003
  Web Magnet - badly written downloader?
2 jmccormac 7:42 pm Feb 8, 2003
  New Inktomi IP
Its a new one for me...
5 Josk 3:20 pm Feb 7, 2003
  New Inktomi Spider...
2 Mtlinfo 3:09 pm Feb 7, 2003
  AOL bot?
198.81.26.* all over my site
2 Giacomo 11:05 am Feb 6, 2003
  Research Machines plc
3 wilderness 3:55 am Feb 6, 2003