Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Sitemap and Robots File

If both have same entry



1:02 pm on Jan 13, 2009 (gmt 0)

5+ Year Member

suppose my XML sitemap contains a URL which i have restricted in robots.txt file, then what will be the result? i know that first crawler will view robots file and will not crawl the restricted URL but, when it reads the sitemap it will find it. will it ignore it or crawl it?

moreover, if i want to restrict the indexing of a URL, then, is it sufficient to not include it in Sitemap or it has to restricted in robots file.


1:26 pm on Jan 13, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Good bots wil ignore it, even if it is in sitemap.

For not indexing the URL, just add it the robots.txt file. You can also change the HTML meta data of your HTML page to none, but not necessary

Google reference here - [googlewebmastercentral.blogspot.com...]

If you use both a robots.txt file and robots meta tags
If the robots.txt and meta tag instructions for a page conflict, Googlebot follows the most restrictive. More specifically:
If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page.
If you allow a page with robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.


4:27 am on Jan 14, 2009 (gmt 0)

5+ Year Member

thanks tomda


3:25 pm on Jan 16, 2009 (gmt 0)

10+ Year Member

Are there still people out there building XML sitemaps?

I have never used one and never had an issue.

A client of mine mentioned to me once that he submitted a sitemap and his site was not indexed 6 months later.

I have never bothered with one since. :)


6:32 am on Jan 19, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Actually, I think Sitemap and Google Webmasters Tools are a must !

Featured Threads

Hot Threads This Week

Hot Threads This Month