- - Search Engines
- -- Alternative Search Engines
- ---- Blekko does not appear to honor ROBOTS.TXT
incrediBILL - 12:39 am on Jan 8, 2011 (gmt 0)
Didn't say blekko didn't honor robots.txt, I said it APPEARS to not honor robots.txt
I can block you in robots.txt yet my site and all it's SEO data still appears.
Plus you display SERPs pulled from 3rd party Yahoo API which at first glance, if you miss the disclaimer it's Yahoo results, would appear you crawled.
Not only that, in the SEO section for WebmasterWorld it claims you crawled, HUH?
|Crawled: 11h ago |
Pages Seen: 7,743
Pages Crawled: 811
Avg Page Length: 0
Avg Page Latency:
Robots: [webmasterworld.com...] (last fetched: 26d ago)
Home Page: [webmasterworld.com...]
So the question is, if robots.txt doesn't really opt-out fully, how the heck do you completely opt-out of Blekko?
No Yahoo API results, no SEO, nada.
Thread source: http://www.webmasterworld.com/alternative_search_engines/4249214.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com