shaunm - 10:33 am on Sep 4, 2012 (gmt 0)

@lucy24
Thank you!

Also: Once the googlebot finds its name in robots.txt, it ignores all other sections. So if you want to block some areas from googlebot, and some areas from all robots, you'll have to say those parts twice.

I have seen this on may websites and had wondered why do they repeat all the files diff spiders such as 'google bot' 'that of yahoo' 'that of alexa', ask etc. So from my above robots.txt, I am just going to remove the 'Noindex' section which as many of you have told is of no use. If I remove that section, then the command 'User-agent: Googlebot' will also get removed and there will be only one command for all the files 'User-agent: *'. It is enough right?

@not2easy
Thanks!

Still, if it shows up in your sitemap they may index it anyway. That is because if you read about the purpose of the sitemap, it is to have a list of the pages you want to have indexed. I found out the hard way a long time ago that you need to only have pages in the sitemap that you do want indexed, because a noindex metatag on the page gets ignored when they find it in the sitemap

This is what makes me learn more about SEO. Thanks for letting me know of that bud!

I am reminded of it again whenever I try to do away with an old page and forget to remove it from the sitemap after I put a noindex metatag on the page.

Out of curiosity, why don't you put a redirect in place?

I submit new sitemaps and still see 404s from pages that have not existed for two years, are not in any current sitemap. I appreciate that I can now mark them as "Fixed" but I know they will be back.

I have the same problems, my website has more than 600000 pages and I am getting 18k server errors through the GWT crawling error section. It shows pages that never existed in my website and whenever I mark them as fixed it again shows up, I fed up with the 'mark as fixed' process.

@Shaddows
Thanks :)
I thought we were talking about the Sitemap, not the ranking factors.