Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Tips to detect what are bad bots in your bot trap
And what are humans clicking bot trap
21 silverbytes 2:02 pm Apr 18, 2007
  New Yahoo bot.
Yahoo-Test/4.0
4 bobothecat 2:11 am Apr 18, 2007
  Faking to be Googlebot
2 sami 2:57 am Apr 17, 2007
  Banning Microsoft URL Control
Best method?
18 bouncybunny 10:13 am Apr 14, 2007
  Jeeves
What they up to?
3 wilderness 1:53 am Apr 12, 2007
  K-Meleon
13 wilderness 11:11 pm Apr 9, 2007
  someone spoofing?
these are some wierd UAs
5 wkitty42 2:11 am Apr 9, 2007
  Songbird/0.1
Botnet, Controlled?
4 blend27 10:44 pm Apr 8, 2007
  RufusBot
"Selects and repackages" web site pages?
15 bouncybunny 6:25 pm Apr 8, 2007
  Human or Bot?
6 Mokita 5:54 pm Apr 8, 2007
  RufusBot
2 writer51 3:05 am Mar 31, 2007
  Microsoft Office Existence Discovery
Appears to be human powered but not sure.
3 GaryK 7:27 pm Mar 26, 2007
  EmeraldShield.com
robots rules don't apply to them
3 koan 11:03 pm Mar 25, 2007
  Deny Morfues [expletive deleted] Scanner
2 TheDuDe 7:32 pm Mar 23, 2007
  guglibot
Anyone heard of this one?
3 malachite 10:24 am Mar 22, 2007
  CazoodleBot
formerly nutch
10 keyplyr 5:21 am Mar 20, 2007
  New MS Search UA?
3 GaryK 1:55 pm Mar 13, 2007
  fembot
Is this new?
3 Mokita 11:01 pm Mar 12, 2007
  Strange user agent
2 fischermx 10:33 pm Mar 12, 2007
  Is it possible ;)
13 wilderness 5:34 pm Mar 6, 2007
  MSN IP's?
8 Brett_Tabke 2:20 pm Mar 6, 2007
  why dont we see spiders in Live chat softwares
sorry for this foolish query
3 benevolent001 3:15 am Mar 5, 2007
  Unvoluntary banning googles sitemap crawler?
What the bot name or ip is?
3 silverbytes 7:52 pm Mar 2, 2007
  Possible Bot or Spammer?
Or is this Live bot?
11 EarleyGirl 9:49 pm Feb 28, 2007
  HoneyNet White Paper : Know your Enemy: Web Application Threats
3 Brett_Tabke 7:36 pm Feb 28, 2007