Msg#: 3825261 posted 1:02 pm on Jan 13, 2009 (gmt 0)
suppose my XML sitemap contains a URL which i have restricted in robots.txt file, then what will be the result? i know that first crawler will view robots file and will not crawl the restricted URL but, when it reads the sitemap it will find it. will it ignore it or crawl it?
moreover, if i want to restrict the indexing of a URL, then, is it sufficient to not include it in Sitemap or it has to restricted in robots file.
If you use both a robots.txt file and robots meta tags If the robots.txt and meta tag instructions for a page conflict, Googlebot follows the most restrictive. More specifically: If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page. If you allow a page with robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.