Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  the DotBot returns
14 lucy24 4:37 pm Oct 12, 2016
  SafeDNSBot
3 keyplyr 11:21 pm Oct 10, 2016
  Pantech
4 keyplyr 8:35 pm Oct 10, 2016
  Drake Holdings LLC
[2] ( 1 2 )
40 Flyby_Knight 12:33 pm Oct 7, 2016
  SimpleScraper
2 keyplyr 11:33 pm Oct 4, 2016
  Lenn\xc3\xa9-Gesamtschule
3 keyplyr 10:58 am Oct 4, 2016
  woobot
3 topr8 11:28 pm Oct 2, 2016
  Internet Archive Python
4 Angonasec 12:41 pm Sept 29, 2016
  Favicon downloader
2 keyplyr 9:31 pm Sept 28, 2016
  another plainclothes bingbot?
5 lucy24 11:58 am Sept 28, 2016
  SimplePie, Feedfetcher & other RSS fetchers
11 keyplyr 9:02 pm Sept 17, 2016
  Yahoo! Slurp
faked UAs
3 keyplyr 10:39 am Sept 5, 2016
  BUbiNG
2 keyplyr 3:18 am Sept 5, 2016
  Melvil Stakhanov
8 keyplyr 3:16 am Sept 5, 2016
  Hello Crawler
6 keyplyr 1:36 pm Sept 3, 2016
  Mozilla-1.1
and other fake mozilla user agents
19 lucy24 4:33 pm Aug 26, 2016
  Botnet-based scans
4 dstiles 2:34 am Aug 19, 2016
  android-app googlequicksearchbox
android-app://com.google.android.googlequicksearchbox
13 dstiles 11:34 am Aug 12, 2016
  eset.com
2 keyplyr 4:26 am Aug 11, 2016
  GuzzleHttp
and other HTTP clients
3 blend27 11:31 pm Aug 8, 2016
  WhatsApp
4 keyplyr 8:06 pm Aug 2, 2016
  UXCrawlerBot
4 keyplyr 8:37 pm Aug 1, 2016
  roboto
3 keyplyr 9:02 pm July 28, 2016
  SocialRankIOBot
5 keyplyr 7:41 pm July 26, 2016
  newspaper
2 keyplyr 9:39 pm July 23, 2016