Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Proposal for robots.txt To Have Greater Flexibility


Receptional - 4:34 pm on Nov 29, 2007 (gmt 0)


Most of the existing commands - even the basic ones - seem to be fail way too often, so I'm not sure more will be better than the sitemaps protocol which seems to be stealing a lead over robots.txt.

The formal rules allow a site to block indexing of individual Web pages, specific directories or the entire site, though some search engines have added their own commands

They were never "formal" rules. They were pretty informal from the off. As I recall it was some long since gone engine that started the protocol and others informally adopted it.

Let's see if anything gets beyond a press release.


Thread source:: http://www.webmasterworld.com/robots_txt/3516185.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com