Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Sosospider
how to stop it?
15 smallcompany 5:49 am Oct 29, 2010
  semrush/whorush
web marketing tools
3 Dijkgraaf 4:39 am Oct 29, 2010
  Which Proxy headers do you capture?
There are so many
3 Clark 11:11 pm Oct 27, 2010
  Fake iPhone and iPad UA requests?
Possibly a botnet
3 jdMorgan 4:18 pm Oct 27, 2010
  RiverglassScanner
4 keyplyr 10:50 pm Oct 26, 2010
  pzncrawl
2 Pfui 10:01 pm Oct 26, 2010
  Lots of Traffic from Rochester, NY
IP addresses located in Rochester, NY either through RoadRunner or Frontier
4 suga 5:26 pm Oct 26, 2010
  Webairinternet2
4 keyplyr 10:53 pm Oct 25, 2010
  Babya Discoverer
4 keyplyr 10:40 pm Oct 25, 2010
  YahooMobile - bot or proxy?
3 dstiles 9:41 pm Oct 25, 2010
  Covenant Eyes
6 caribguy 2:16 am Oct 24, 2010
  flipboard
4 Pfui 5:39 pm Oct 23, 2010
  SiteBot
Should SiteBot be blocked?
8 grandma_genie 9:20 am Oct 23, 2010
  sogou.com IP address range
3 JAB_Creations 11:55 pm Oct 20, 2010
  Toms Spider 0.3
3 Dijkgraaf 6:24 am Oct 19, 2010
  Mozilla/5.0 (compatible; askpeter bot/5.0; +http://www.askpeter.info)
4 Dijkgraaf 4:47 am Oct 19, 2010
  ResearchProject
13 Pfui 8:47 pm Oct 18, 2010
  Jomjaibot
2 Dijkgraaf 5:46 am Oct 18, 2010
  Jaxified Crawler
9 keyplyr 2:11 am Oct 18, 2010
  x-www-form-urlencoded
3 topr8 8:08 am Oct 17, 2010
  webmoney.advisor/Nutch-1.0
Russian online payments?
8 Dijkgraaf 6:11 pm Oct 14, 2010
  doondbot
2 keyplyr 5:30 am Oct 12, 2010
  Findshare bot
Found this Russian? bot hitting my logs today.
12 grandma_genie 12:14 am Oct 12, 2010
  Yahoo Nokia Symbian crawler
Is this a Yahoo crawler for mobile aps?
3 grandma_genie 1:33 am Oct 9, 2010
  Catchbot
Fetches robots.txt now
3 Dijkgraaf 8:44 am Oct 7, 2010