Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Google's Current Specifications for Robots Directives


graeme_p - 8:04 am on Nov 26, 2010 (gmt 0)


@tangor, how are Google trying to bend the standard? As far as I can see the differences are the addition of the wildcard character (which all the major search engines support), accepting UTF-8 (with option BOM), the addition of the sitemap directive, and the limit on the number of redirects. All should be backward compatible under any reasonable circumstances. Am I missing anything that matters?


Thread source:: http://www.webmasterworld.com/robots_txt/4234719.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com