Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Proposal for robots.txt To Have Greater Flexibility


mikedee - 6:29 pm on Nov 29, 2007 (gmt 0)


This seems aimed at large online newspapers etc. There is nothing here for the average webmaster.

From my quick overview of the spec, I could only see tags to tell search engines how much of your content they can show in snippets and images. It looks like they are trying to limit how much of their content can be shown in the search results. Maybe one day they intend for this to be legally binding and some sort of override of copyright fair-use.

robots.txt + sitemap is fine for 99% of websites I think. robots.txt could be tweaked, but I think the big search engines will dictate that through slow evolution (like the Sitemap tag)


Thread source:: http://www.webmasterworld.com/robots_txt/3516185.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com