- - Search Engines
- -- Alternative Search Engines
- ---- Blekko does not appear to honor ROBOTS.TXT
incrediBILL - 3:09 am on Jan 5, 2011 (gmt 0)
Spun off from this thread about blekko and NOARCHIVE:
I discovered WebmasterWorld was on blekko, and appeared to honor NOARCHIVE for the pages it showed of WebmasterWorld.
|Go check out WebmasterWorld's cache pages on blekko: |
I see snippets and when I click cache "Error: No content" so for WebmasterWorld it appears to be implemented to support NOARCHIVE while maintaining the snippets in the index.
Found out WebmasterWorld blocks blekko with robots.txt, blekko is banned from crawling WebmasterWorld, and the listings are being pulled from some 3d party SE API
How do you opt-out of blekko?
Apparently you can't.
Thread source: http://www.webmasterworld.com/alternative_search_engines/4249214.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com