homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum


 2:32 pm on Aug 27, 2010 (gmt 0)

I understand that pages using the metatag noindex allow for crawling, following links and accrual of page rank but do not get indexed. However, I see my site growing in the index rapidly because of these pages -mostly worthless-. Does anyone know how it works? Googlebot crawls them and then drops them from the index? I was expecting these pages never to be counted in the "About xx results" that come up in a "site" search command. Right now, the headers have the "noindex,follow". Should I switch to "noindex,nofollow" to stop the crawling and prevent the indexing completely?



 1:04 am on Aug 28, 2010 (gmt 0)

are you seeing just the noindexed urls in the index or actual content from those pages?


 4:55 am on Aug 28, 2010 (gmt 0)

It's probably too early to draw any conclusion. Site is fairly new and I made changes as it got crawled and indexed. The noindex pages that have been indexed do have content showing in cache but taking a closer look I also see that they were crawled before I switched the meta to noindex. That explains it. Planning ahead is essential!

I do not see the pages that have been crawled in the last few days, after I made the change. I imagine the ones indexed will get drop after they are crawled again. Sorry for wasting your time.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved