homepage Welcome to WebmasterWorld Guest from 54.227.171.163
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Sitemap and robot.txt
spuddy

5+ Year Member



 
Msg#: 3253243 posted 11:19 am on Feb 15, 2007 (gmt 0)

Hi there,

Our google sitemap is crawling everything on the server, would the robots.txt file stop it from indexing these pages?

Am i right in saying that the sitemap lets google know every page that you have on your server, and that the robots.txt file will tell it which out of those it has found we do not want it to index?

Do the two files work alongside eachother?

We have different directories on the server that we need to keep on there but we do not want included in googles index. They are not linked from anywhere so this was never an issue until we submitted a google sitemap, and now google has found them all.

If i add these pages to the robots.txt file will it stop google indexing them?

Thanks

Spuddy

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3253243 posted 1:08 pm on Feb 15, 2007 (gmt 0)

If i add these pages to the robots.txt file will it stop google indexing them?

yes, robots.txt will override a sitemap. a sitemap is a simple way to tell google that pages exist and they should be crawled. robots.txt is a way to control googlebot.

you can use robots.txt to stop googlebot from accessing certain areas of your site. of course if you block a page with robots.txt then it doesn't make much sense to add it to your sitemap.

cheers

spuddy

5+ Year Member



 
Msg#: 3253243 posted 3:53 pm on Feb 15, 2007 (gmt 0)

Ok thanks goodroi

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved