Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Google Index robots.txt


phranque - 3:52 pm on Aug 25, 2012 (gmt 0)


a sitemap is simply a url discovery mechanism.

reqardless of how a crawler has discovered a url, the robots.txt file is consulted to see if the url matches an excluded pattern before requesting the url.

GWT should report this as crawl "error".

i wouldn't recommend putting excluded urls in a sitemap.xml file.


Thread source:: http://www.webmasterworld.com/robots_txt/4487907.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com