maximillianos - 4:48 pm on Feb 9, 2010 (gmt 0)
I've been working on a monitoring script that reports on the top bots crawling my site at any given moment. During my testing I've noted a few bots that are legit, but from other countries.
My question, should I block these bots? Yandex for instance is a Russian bot for a Russian search engine/portal. My site is US based, and my traffic is 99% US based. So I'm not getting much benefit from Russian traffic. Does this tell me I should just block them and save my processor time/bandwidth being eaten up by them?
Currently they fall in 3rd in regards to how much they crawl my site only behind Google and Yahoo. We are talking tens of thousands of pages a day they crawl.
What advice would you give me? At first I thought why not just leave it, it is more exposure for my site. But then I started thinking maybe it was pointless?
I guess the same question goes for all those prototype search bots who are trying to make a name for themselves. They typically don't crawl many pages, but they are always on the site.
Thanks for any tips.