Msg#: 4249214 posted 10:17 am on Jan 5, 2011 (gmt 0)
Didn't have time to start a new topic yesterday, but after reading the blekko thread also discovered that at least one of my sites (not just home page which could be done without crawling) appears to be in blekko despite the fact they've never been whitelisted (i.e. blekko would be receiving the bog standard disallow all robots.txt).
Msg#: 4249214 posted 6:32 pm on Jan 6, 2011 (gmt 0)
blekko strictly honors robots.txt. We do not crawl webmasterworld.com, as evidenced by the lack of a cached page in our serps.
Like Google and Bing, however, we still include serps for sites which we can't crawl if we have enough inbound anchortext material. Snippets can be obtained from anchortext, dmoz or other metadata without crawling the site.