dstiles - 8:01 pm on Feb 9, 2010 (gmt 0)

I allow Yandex to crawl my server's sites here (UK). I see the occasional hit from the SE but not enough to worry about. I originally allowed it for a client with a Russian-facing site, now no longer up.

Taking only two sites' stats for the past six weeks, top crawl on both sites is msn, slurp and yandex followed by "unknown" on one site and vagabondo on another. Goog comes in very low on the hits (48 of 600 total and 125 of 1500 total).

I recently ran an excercise tying in yandex crawl IPs with their bot's User-Agent. Very messy. Far fewer actual IPs than MS but fragmented across more Class C's.

I'm working through several other bots at the moment, especially those used by meta engines, ensuring that when everyone drops google there is still something left to send traffic to my customers. :)