Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Does IP Range Block Via WHM Blacklist Block Same IP Ranges at cPanel
Does IP Block at WHM BruteForce Level Filter Down to Unique Domains on VPS?
12 Webwork 10:40 pm Jan 14, 2014
  BRIC nations
Eating Bricks
4 Angonasec 7:27 am Jan 13, 2014
  sputnik
6 lucy24 7:52 pm Jan 12, 2014
  Zeerch
9 Angonasec 6:30 pm Jan 11, 2014
  Rampant of Redmond
Shivering my Timbers
7 Angonasec 12:52 pm Jan 10, 2014
  112.0.0.0/8
7 keyplyr 9:43 pm Jan 7, 2014
  UNTRUSTED in Nokia User Agent
[2] ( 1 2 )
45 lucy24 11:41 am Jan 4, 2014
  thunderstone
yes, them again
3 lucy24 6:28 pm Jan 1, 2014
  Come on Yahoo. Really?
7 bobothecat2 8:38 pm Dec 27, 2013
  archive.org bot
bad bot
7 keyplyr 7:20 am Dec 20, 2013
  Google Test-Bot: Google-Test2
11 doc_z 8:28 pm Dec 19, 2013
  Google? Is that you?
13 lucy24 10:33 pm Dec 18, 2013
  Is there a Browser. . .
10 wilderness 12:53 am Dec 16, 2013
  TECH UPDATE: Javascript Makes Dumb Bots Obsolete
Javascript Must Be Crawled For Scrapers To Survive
3 incrediBILL 10:12 pm Dec 15, 2013
  the plainclothes bingbot
8 lucy24 8:03 pm Dec 14, 2013
  Yahoo:LinkExpander:Slingstone
7 keyplyr 2:06 pm Dec 13, 2013
  Google Translate
7 wilderness 2:43 am Dec 11, 2013
  YaBrowser
15 lucy24 10:04 am Dec 10, 2013
  New stupid referer spammer
Referer spam via agent string (mistyped)
9 yaimapitu 2:58 pm Dec 8, 2013
  rarely used
12 lucy24 9:22 am Dec 1, 2013
  New Scraping Tool
9 keyplyr 9:07 am Nov 28, 2013
  Yahoo phone home
19 lucy24 8:07 pm Nov 23, 2013
  Amazon aws, dedicatedcpanel, serverloft Issues
These domains are dedicatedly screwing my site - Need Help.
10 Future 7:59 pm Nov 23, 2013
  TECH UPDATE: Bots as Browsers Using JavaScript
The Bot Detection Game Has Changed[2] ( 1 2 )
48 incrediBILL 9:18 pm Nov 11, 2013
  google-xrawler
Unknown User agent "google-xrawler"
15 DutchTheWiz 7:51 pm Nov 7, 2013