Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Google's Current Specifications for Robots Directives

Samizdata - 1:07 am on Nov 25, 2010 (gmt 0)


Note that none of this applies to the new Google Web Preview bot.

It only exists to bypass the robots.txt restrictions placed on Googlebot.

...

Thread source:: http://www.webmasterworld.com/robots_txt/4234719.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com