true_INFP - 9:57 am on Jan 2, 2011 (gmt 0)
What is Blekko?
What worries me more are major sites like the Wayback Machine archive.org ignoring the noarchive tag...
By the way, we considered white-listing bots in robots.txt (thus banning all unknown robots). However, we concluded that we would ban many important search engines in countries we know nothing about. Individually, none of those search engines shows as a significant traffic source in your stats, but together they are an important and strong source. You may know the Russian Yandex and Chinese Baidu, but there are dozens of smaller countries, each with their own popular search engine. That is why we do not white-list, but black-list bots in robots.txt.