Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  open proxies
7 wilderness 12:12 pm May 3, 2012
  MojeekBot
14 wilderness 2:16 pm May 2, 2012
  fatlens.com using Googlebot agents?
5 brokaddr 1:19 pm May 2, 2012
  Besthosting
3 keyplyr 9:46 pm Apr 29, 2012
  EGIHosting
4 keyplyr 2:20 pm Apr 28, 2012
  WeblexBot
8 keyplyr 7:26 pm Apr 27, 2012
  *missing* - user agent from Yahoo, or spoof?
7 brokaddr 3:13 am Apr 27, 2012
  LitlrBot
mobile SE
2 keyplyr 1:16 am Apr 27, 2012
  AWS Ireland?
12 incrediBILL 5:59 am Apr 22, 2012
  more colo's
5 wilderness 1:56 am Apr 22, 2012
  Multi-National Opera 8 invasion
11 incrediBILL 8:49 am Apr 21, 2012
  Google search without keywords valid
6 motorhaven 5:06 am Apr 21, 2012
  HEAD checks
3 wilderness 3:56 am Apr 20, 2012
  AskTb redirect
4 wilderness 3:40 am Apr 20, 2012
  Speakeasy
4 wilderness 9:53 pm Apr 19, 2012
  Electricfiber.net
Nuance/Tegic Linguistics Team
4 keyplyr 9:01 pm Apr 17, 2012
  What is bot[+:,\.\;\/\\-]?
7 ichthyous 10:17 pm Apr 16, 2012
  Jakarta Commons
3 wilderness 8:24 pm Apr 16, 2012
  ia archiver
random hits from China
19 dstiles 9:14 pm Apr 15, 2012
  EventMachine
3 wilderness 9:05 pm Apr 14, 2012
  "\"Mozilla/5.0"
8 keyplyr 10:11 am Apr 13, 2012
  PacketExchange
6 wilderness 8:08 pm Apr 11, 2012
  Googlebot Has Ears
and they seem to like good music
13 Samizdata 8:17 pm Apr 10, 2012
  184
9 wilderness 4:26 pm Apr 10, 2012
  ChinaCache
7 wilderness 11:40 pm Apr 6, 2012