Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
  Bots start looking for back.zip, backup.zip, etc.
Get rid of those if you have them in root.
3 SumGuy 12:32 pm Apr 10, 2019
  Google-speakr
6 phranque 9:23 pm Apr 5, 2019
  User Agent is Microsoft Office something or other
5 SumGuy 8:29 pm Apr 4, 2019
  Google 34
2 lucy24 1:48 am Apr 2, 2019
  Facebook cortex
8 lucy24 1:21 am Apr 2, 2019
  35.244.0.0/14 - Google or Merit (U-Mich) ?
4 SumGuy 6:23 pm Apr 1, 2019
  Getting lots of hits from this Cloudsigma. what's it?
3 born2run 5:58 am Mar 24, 2019
  Apple applebot increasing activity
Increasing hits to interior web pages
4 SumGuy 4:32 am Mar 18, 2019
  Google polyfill
8 lucy24 3:34 am Feb 25, 2019
  104.199.128.134
7 andrew_thom 2:19 am Feb 24, 2019
  Skotina via TOR
After so many years, scrapers are still there...
3 blend27 11:49 pm Feb 13, 2019
  URLTester
3 lucy24 6:07 am Feb 3, 2019
  Symantec from **.broadband.digiweb.ie
The range belongs to Symantec Corporation
7 blend27 10:09 pm Jan 28, 2019
  Microsoft and 13.64.0.0/11
Legit bing-bots? BingLocalSearch?
14 SumGuy 7:04 pm Jan 21, 2019
  Discordbot
3 keyplyr 2:40 am Jan 14, 2019
  iCab/5.8.4
2 TorontoBoy 1:03 am Jan 13, 2019
  7Siters
3 TorontoBoy 5:34 pm Dec 24, 2018
  New Range for COCCOC bot?
4 blend27 7:34 pm Dec 15, 2018
  facebookexternalhit
8 lucy24 2:54 pm Dec 10, 2018
  Laserlikebot/0.1
2 TorontoBoy 4:55 am Dec 5, 2018
  BigSkyBot 2.06
Simple site parsing agent for niche search index
2 mcneely 1:57 am Nov 22, 2018
  Falkon
5 lucy24 9:38 am Oct 31, 2018
  OutclicksBot
new UA, new range
12 keyplyr 9:04 pm Oct 30, 2018
  PocketImageCache
2 keyplyr 3:11 pm Oct 24, 2018
  Aranea Web Corpora Project
3 keyplyr 6:41 am Oct 24, 2018