Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  DNS-Digger-Explorer
Reprise of 2006 report in WebmasterWorld
5 dstiles 3:17 pm Nov 11, 2009
  Yandex (redux)
8 Pfui 4:19 am Nov 10, 2009
  COAST WebMaster
3 Pfui 3:22 am Nov 10, 2009
  http://arachnode.net 1.2
2 GaryK 11:02 pm Nov 9, 2009
  Ayna
Arabic search engine
2 GaryK 10:11 pm Nov 9, 2009
  Volume Drive
2 keyplyr 10:05 pm Nov 8, 2009
  New stuff from Yahoo
8 GaryK 2:27 am Nov 7, 2009
  Lynxy/6.6.6dev.8 libwww-FM/3.14159FM
2 GaryK 2:00 am Nov 6, 2009
  Mp3Bot
3 keyplyr 11:14 pm Nov 3, 2009
  atraxbot
using new UA
3 keyplyr 7:06 pm Nov 3, 2009
  MSN Translators - referer Formats - 65.55.177.205
windowslivetranslator.com; translator.live.com; microsofttranslator.com
5 Pfui 3:58 am Nov 3, 2009
  WukongBot regularly hits my site
6 greenTree 10:21 am Oct 30, 2009
  Jigsaw
2 wilderness 8:25 pm Oct 29, 2009
  Mozilla/5.0 (compatible; Twitturls; http://twitturls.com)
6 GaryK 11:24 am Oct 27, 2009
  Getting caught up
Sorry I've been away for awhile
5 GaryK 1:47 am Oct 24, 2009
  AAfter.com Crawler
4 keyplyr 11:31 pm Oct 23, 2009
  Y!j-bro
is this legit?
4 GaryK 10:11 pm Oct 23, 2009
  Oracle Ultra Search
2 Pfui 2:58 pm Oct 23, 2009
  Isara/Isara-1.0
5 keyplyr 2:51 pm Oct 23, 2009
  google.com -- spoof? spider? botnet zombie? employee?
10 Pfui 10:45 pm Oct 20, 2009
  Comcast Launches Anti-Botnet Initiative "Constant Guard"
Infected Owners Will Be Warned
28 incrediBILL 3:05 am Oct 20, 2009
  NeoDownloader
image ripper
2 dstiles 4:19 pm Oct 19, 2009
  Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows)
2 smallcompany 5:57 am Oct 17, 2009
  SurveyBot/2.3 (D o m a i n T o o l s)
Revisited. Republishing (c) Content. Re-Exposed.
3 Pfui 2:53 am Oct 16, 2009
  Real-time checking IPs
scrapers are hitting me hard
2 SEOPTI 2:01 pm Oct 15, 2009