- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Google's Current Specifications for Robots Directives
tangor - 5:52 am on Nov 25, 2010 (gmt 0)
"robots.txt"? We don't need no steenkin' robots.txt
Except for those that do honor whitelisting, etc. Google, as usual, seeks to bend standards to their standard. And, sad to say, the sheeple webmasters will follow.
Thread source: http://www.webmasterworld.com/robots_txt/4234719.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com