Page is a not externally linkable
- Search Engines
-- Alternative Search Engines
---- SearchMe is pretty cool


incrediBILL - 5:51 pm on Sep 28, 2008 (gmt 0)


I'm curious how you manage to have complete ( text and all images ) pages from one of my image based sites ..inspite of the fact that the image folders are denied

That's because they're part search engine and part screen shot.

Robots.txt applies to search engines but doesn't apply to browsers which has put them in this murky fuzzy gray area where I would agree with you that the screen shot tool should also honor robots.txt since the purpose of the screen shot in SearchMe is similar to Google's CACHE pages which do honor the NOARCHIVE directive.


Thread source:: http://www.webmasterworld.com/alternative_search_engines/3751337.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com