Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Index Page Removal, Serious Problem Please Help


phranque - 2:16 pm on May 26, 2011 (gmt 0)


HTTP/1.1 Status Code Definitions:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html [w3.org]
HTTP/1.1 Header Field Definitions:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html [w3.org]

although i would probably be most concerned with the Link: rel="canonical" header and the "X-Meta-" response headers (which aren't listed in the above reference and are typically defined in the document <head>), especially the meta robots header.
or the X-Robots-Tag header.

use the Live HTTP Headers plugin for FF or your favorite "status code/server header checker tool".


Thread source:: http://www.webmasterworld.com/robots_txt/4317407.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com