- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- To Block or Not To Block That is The Question
dstiles - 7:26 pm on Jan 26, 2014 (gmt 0)
You will see several comments over the past few years that say, "Block all server farms but drill holes for useful and genuine bots".
In my view the latter includes yandex, which is far less intrusive than google or bing. I allow the RU version since I suspect they may share results with the US version.
Thread source: http://www.webmasterworld.com/search_engine_spiders/4640214.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com