Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  alexa site audit
2 keyplyr 12:58 am Feb 18, 2017
  German Wikipedia Broken Weblinks Bot
tools.wmflabs.org
4 jonasjacek 12:27 pm Feb 17, 2017
  Typo3
7 keyplyr 5:43 am Feb 16, 2017
  DER-bot
8 jonasjacek 9:10 am Feb 15, 2017
  Microsoft Office Protocol Discovery
5 lucy24 8:02 am Feb 8, 2017
  Whoiswebsitebot
3 keyplyr 3:05 am Feb 2, 2017
  gutsy
Ubuntu/7.10
3 keyplyr 2:37 am Jan 28, 2017
  spyonweb
2 jonasjacek 8:35 am Jan 26, 2017
  eCatch
3 keyplyr 4:29 am Jan 26, 2017
  TT SNBReact Bot
3 keyplyr 3:01 am Jan 25, 2017
  pshtt, https scanning
3 keyplyr 5:27 am Jan 19, 2017
  Leaseweb/Comcast connection?
4 LifeinAsia 9:17 pm Jan 16, 2017
  node-fetch
4 keyplyr 3:03 am Jan 13, 2017
  BoogleBot
17 jonasjacek 4:44 pm Jan 9, 2017
  Server Farms - June 2016
Continuation of the discussion of hosting company IP ranges[3] ( 1 2 3 )
88 Ocean10000 8:58 pm Jan 8, 2017
  redback
3 jonasjacek 7:40 pm Jan 6, 2017
  testitest
4 keyplyr 5:10 pm Jan 6, 2017
  MS impersonating FB
7 keyplyr 8:39 am Jan 6, 2017
  brandwatch magpiecrawler
5 lucy24 5:32 am Jan 6, 2017
  now what's Yandex up to?
4 lucy24 12:52 pm Jan 3, 2017
  HubSpot Marketing Grader
5 keyplyr 9:13 pm Jan 1, 2017
  Baiduspider Crawl Ranges
verified IP ranges
16 keyplyr 2:13 am Dec 31, 2016
  UniversalFeedParser
3 keyplyr 5:19 am Dec 23, 2016
  Screaming Frog SEO Spider
8 keyplyr 9:50 pm Dec 22, 2016
  spbot
3 keyplyr 5:03 am Dec 22, 2016