Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: bakedjake
Go check out WebmasterWorld's cache pages on blekko:
I see snippets and when I click cache "Error: No content" so for WebmasterWorld it appears to be implemented to support NOARCHIVE while maintaining the snippets in the index.
Crawled: 11h ago
Pages Seen: 7,743
Pages Crawled: 811
Avg Page Length: 0
Avg Page Latency:
Robots: [webmasterworld.com...] (last fetched: 26d ago)
Home Page: [webmasterworld.com...]
So the question is, if robots.txt doesn't really opt-out fully, how the heck do you completely opt-out of Blekko?
No Yahoo API results, no SEO, nada.