homepage Welcome to WebmasterWorld Guest from 54.204.217.249
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Sitemap and Robots File
If both have same entry
vicky




msg:3825263
 1:02 pm on Jan 13, 2009 (gmt 0)

suppose my XML sitemap contains a URL which i have restricted in robots.txt file, then what will be the result? i know that first crawler will view robots file and will not crawl the restricted URL but, when it reads the sitemap it will find it. will it ignore it or crawl it?

moreover, if i want to restrict the indexing of a URL, then, is it sufficient to not include it in Sitemap or it has to restricted in robots file.

 

tomda




msg:3825271
 1:26 pm on Jan 13, 2009 (gmt 0)

Good bots wil ignore it, even if it is in sitemap.

For not indexing the URL, just add it the robots.txt file. You can also change the HTML meta data of your HTML page to none, but not necessary

Google reference here - [googlewebmastercentral.blogspot.com...]

If you use both a robots.txt file and robots meta tags
If the robots.txt and meta tag instructions for a page conflict, Googlebot follows the most restrictive. More specifically:
If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page.
If you allow a page with robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.

vicky




msg:3825852
 4:27 am on Jan 14, 2009 (gmt 0)

thanks tomda

stakaman




msg:3827876
 3:25 pm on Jan 16, 2009 (gmt 0)

Are there still people out there building XML sitemaps?

I have never used one and never had an issue.

A client of mine mentioned to me once that he submitted a sitemap and his site was not indexed 6 months later.

I have never bothered with one since. :)

tomda




msg:3829317
 6:32 am on Jan 19, 2009 (gmt 0)

Actually, I think Sitemap and Google Webmasters Tools are a must !

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved