homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Sitemap and Robots File
If both have same entry

 1:02 pm on Jan 13, 2009 (gmt 0)

suppose my XML sitemap contains a URL which i have restricted in robots.txt file, then what will be the result? i know that first crawler will view robots file and will not crawl the restricted URL but, when it reads the sitemap it will find it. will it ignore it or crawl it?

moreover, if i want to restrict the indexing of a URL, then, is it sufficient to not include it in Sitemap or it has to restricted in robots file.



 1:26 pm on Jan 13, 2009 (gmt 0)

Good bots wil ignore it, even if it is in sitemap.

For not indexing the URL, just add it the robots.txt file. You can also change the HTML meta data of your HTML page to none, but not necessary

Google reference here - [googlewebmastercentral.blogspot.com...]

If you use both a robots.txt file and robots meta tags
If the robots.txt and meta tag instructions for a page conflict, Googlebot follows the most restrictive. More specifically:
If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page.
If you allow a page with robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.


 4:27 am on Jan 14, 2009 (gmt 0)

thanks tomda


 3:25 pm on Jan 16, 2009 (gmt 0)

Are there still people out there building XML sitemaps?

I have never used one and never had an issue.

A client of mine mentioned to me once that he submitted a sitemap and his site was not indexed 6 months later.

I have never bothered with one since. :)


 6:32 am on Jan 19, 2009 (gmt 0)

Actually, I think Sitemap and Google Webmasters Tools are a must !

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved