Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Why is Tumblr crawling from ThePlanet?
5 GaryK 7:45 pm May 6, 2009
  Google Wireless Transcoder
74.125.74.nnn acting badly
8 Umbra 4:31 pm May 6, 2009
  Scraper thread on WebmasterWorld
3 incrediBILL 6:30 pm Apr 30, 2009
  MSN stealth 'bot?
MSIE 7 User-agent
5 jdMorgan 10:52 pm Apr 29, 2009
  New junk from Yahoo
7 GaryK 1:41 am Apr 26, 2009
  Foxy in User-Agent
10 dstiles 11:05 pm Apr 24, 2009
  Question about an IE UA
3 GaryK 8:15 pm Apr 24, 2009
  bnf.fr bot
Digital National Library of France
9 keyplyr 9:34 am Apr 23, 2009
  Diffbot
I sent this packing a few months ago. Now it's back with a new UA
5 GaryK 5:06 pm Apr 22, 2009
  Chat Catcher
3 GaryK 3:40 pm Apr 21, 2009
  Some Opera Mobile user agents
10 GaryK 2:39 am Apr 21, 2009
  fake Googlebot?
24 zerillos 2:10 am Apr 21, 2009
  GoogleToolbar and favicon
asking for it everywhere
12 smallcompany 10:59 am Apr 20, 2009
  Daumoa, Korean bot
disobedient and weird UA
4 Mokita 3:52 am Apr 19, 2009
  Bad-Neighborhood Link Analyzer
26 GaryK 9:27 pm Apr 16, 2009
  Unreal
11 wilderness 11:38 pm Apr 14, 2009
  Next Generation Search Engine
I'm seeing a lot of these lately
3 GaryK 5:10 pm Apr 13, 2009
  Strange MSN 2.b Bot Behavior.Hacker?
Strange MSN 2.b Bot Behavior
2 Lain_se 2:59 pm Apr 13, 2009
  Microsoft crawl, no bot UA
13 keyplyr 10:34 am Apr 12, 2009
  GoogleBot Declaring itself in my tracker stats?
never before have I seen this
3 Kristos 11:35 pm Apr 11, 2009
  Comtorbot
2 keyplyr 5:35 pm Apr 5, 2009
  Complete List of Yahoo Crawlers IPs addresses to block them !
How to definitively block yahoo slurp bots?
2 oliondor 2:57 pm Apr 4, 2009
  Hailoo Search
Hailoo? Hailoo? Anybody Home?
17 incrediBILL 8:35 am Apr 4, 2009
  Shelob
new origin
2 keyplyr 10:23 am Apr 1, 2009
  DKIMRepBot
dkim-reputation.org
10 dstiles 9:21 pm Mar 30, 2009