Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Siteimprove
5 keyplyr 10:32 am Mar 13, 2018
  Seeing GoogleBot on 50.234.128.0/17 (Comcast) ?
4 SumGuy 10:04 am Mar 13, 2018
  Check out this blank UA visitor.
7 born2run 3:08 am Mar 9, 2018
  Streamline3Bot
2 keyplyr 4:40 pm Mar 8, 2018
  Can anyone give me a list of UAs I should always ban?
9 born2run 9:56 pm Mar 7, 2018
  another day, another Yandex
7 lucy24 1:16 am Feb 27, 2018
  Symfony BrowserKit
4 keyplyr 11:49 pm Feb 23, 2018
  HyScore
2 keyplyr 3:58 am Feb 22, 2018
  Chromeless
9 lucy24 12:27 am Feb 17, 2018
  mediawords bot
3 keyplyr 9:20 pm Feb 16, 2018
  Oh internet it's 2018 and still . .
Bad Behavior by Bots
2 Webwork 8:48 pm Feb 13, 2018
  TinEye-bot
2 lucy24 11:57 pm Jan 28, 2018
  LittleScraper
3 keyplyr 12:53 am Jan 28, 2018
  HatenaBookmark
more UAs
2 keyplyr 9:02 pm Jan 24, 2018
  Sonic the robot
4 lucy24 5:50 am Jan 20, 2018
  Spoofed robots.txt requester
Single IP range
2 Ilsevel 7:10 pm Jan 17, 2018
  Initial double-quote symbol
5 dstiles 3:21 pm Jan 16, 2018
  glindahl-cocrawler
10 keyplyr 7:08 am Jan 16, 2018
  Windows 200X
Referrer spam pattern
10 Ilsevel 4:20 am Jan 16, 2018
  uCrawler
4 keyplyr 10:03 am Jan 14, 2018
  SleipnirSiteUpdates
3 lucy24 2:20 pm Jan 12, 2018
  Discourse Forum Onebox
3 keyplyr 11:30 pm Jan 8, 2018
  DnyzBot
3 keyplyr 9:56 pm Jan 4, 2018
  Server Farms - Jan 2017
Continuation of the discussion of hosting company IP ranges[3] ( 1 2 3 )
65 Ocean10000 9:44 pm Jan 3, 2018
  LightspeedSystemsCrawler
new UA
2 keyplyr 9:10 pm Dec 27, 2017