Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Google's Current Specifications for Robots Directives
Samizdata - 1:07 am on Nov 25, 2010 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/4234719.htm
Note that none of this applies to the new Google Web Preview bot.
It only exists to bypass the robots.txt restrictions placed on Googlebot.
Brought to you by WebmasterWorld: http://www.webmasterworld.com