-- Search Engine Spider and User Agent Identification
---- To Block or Not To Block That is The Question
trintragula - 12:44 am on Jan 29, 2014 (gmt 0)
@trintragula - Possibly a bad agent spoofing the Yandex UA?
Always possible, but not on the event that first got my attention: 22.214.171.124/24 AS13238 YANDEX Yandex LLC. They were taking about 10x the number of pages that google normally averages per day, and were keeping it up. Baidu were also fairly leadfooted, and unlike Yandex they do seem to ignore robots.txt sometimes. I don't block by country but I get very few forum members from Russia or China, so I've not whitelisted those search engines.