So I did some analysing of my accesslogs and found the following bots (UAs listed) hitting my site most often:
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; ProjectShield-UrlCheck; +http://g.co/projectshield)
Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b13pre) Gecko/20110307 Firefox/4.0b13
There's also hits from the following WOW64:
Mozilla/5.0 (Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0
Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
For now, I'm blocking UAs - ahrefs, Yandex & SemrushBot any reason why I shouldn't be blocking these?
Also should I just block this UA WOW64 as well? Anyone with similar experience? Please advise. Thanks!
