Welcome to WebmasterWorld Guest from 50.19.34.234

Forum Moderators: goodroi

Message Too Old, No Replies

Sitemap and robot.txt

     

spuddy

11:19 am on Feb 15, 2007 (gmt 0)

5+ Year Member



Hi there,

Our google sitemap is crawling everything on the server, would the robots.txt file stop it from indexing these pages?

Am i right in saying that the sitemap lets google know every page that you have on your server, and that the robots.txt file will tell it which out of those it has found we do not want it to index?

Do the two files work alongside eachother?

We have different directories on the server that we need to keep on there but we do not want included in googles index. They are not linked from anywhere so this was never an issue until we submitted a google sitemap, and now google has found them all.

If i add these pages to the robots.txt file will it stop google indexing them?

Thanks

Spuddy

goodroi

1:08 pm on Feb 15, 2007 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



If i add these pages to the robots.txt file will it stop google indexing them?

yes, robots.txt will override a sitemap. a sitemap is a simple way to tell google that pages exist and they should be crawled. robots.txt is a way to control googlebot.

you can use robots.txt to stop googlebot from accessing certain areas of your site. of course if you block a page with robots.txt then it doesn't make much sense to add it to your sitemap.

cheers

spuddy

3:53 pm on Feb 15, 2007 (gmt 0)

5+ Year Member



Ok thanks goodroi
 

Featured Threads

Hot Threads This Week

Hot Threads This Month