Welcome to WebmasterWorld Guest from 54.227.82.149

Forum Moderators: goodroi

Message Too Old, No Replies

Sitemap and Robots File

If both have same entry

     

vicky

1:02 pm on Jan 13, 2009 (gmt 0)

5+ Year Member



suppose my XML sitemap contains a URL which i have restricted in robots.txt file, then what will be the result? i know that first crawler will view robots file and will not crawl the restricted URL but, when it reads the sitemap it will find it. will it ignore it or crawl it?

moreover, if i want to restrict the indexing of a URL, then, is it sufficient to not include it in Sitemap or it has to restricted in robots file.

tomda

1:26 pm on Jan 13, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Good bots wil ignore it, even if it is in sitemap.

For not indexing the URL, just add it the robots.txt file. You can also change the HTML meta data of your HTML page to none, but not necessary

Google reference here - [googlewebmastercentral.blogspot.com...]

If you use both a robots.txt file and robots meta tags
If the robots.txt and meta tag instructions for a page conflict, Googlebot follows the most restrictive. More specifically:
If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page.
If you allow a page with robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.

vicky

4:27 am on Jan 14, 2009 (gmt 0)

5+ Year Member



thanks tomda

stakaman

3:25 pm on Jan 16, 2009 (gmt 0)

10+ Year Member



Are there still people out there building XML sitemaps?

I have never used one and never had an issue.

A client of mine mentioned to me once that he submitted a sitemap and his site was not indexed 6 months later.

I have never bothered with one since. :)

tomda

6:32 am on Jan 19, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Actually, I think Sitemap and Google Webmasters Tools are a must !
 

Featured Threads

Hot Threads This Week

Hot Threads This Month