incrediBILL - 12:39 am on Jan 8, 2011 (gmt 0)
Didn't say blekko didn't honor robots.txt, I said it APPEARS to not honor robots.txt
I can block you in robots.txt yet my site and all it's SEO data still appears.
Plus you display SERPs pulled from 3rd party Yahoo API which at first glance, if you miss the disclaimer it's Yahoo results, would appear you crawled.
Not only that, in the SEO section for WebmasterWorld it claims you crawled, HUH?
Crawled: 11h ago
Pages Seen: 7,743
Pages Crawled: 811
Avg Page Length: 0
Avg Page Latency:
Robots: [webmasterworld.com...] (last fetched: 26d ago)
Home Page: [webmasterworld.com...]
So the question is, if robots.txt doesn't really opt-out fully, how the heck do you completely opt-out of Blekko?
No Yahoo API results, no SEO, nada.