homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Disallowing Indexed files in Google

 9:48 pm on Dec 20, 2006 (gmt 0)

Lets say I have a few hundred paged in regular index and few thousand in Supplemental index. I dont need Google to index those page, is it ok to put them in robots.txt (with wildcards).

If so what will happen to those in regular index? will the drop out immediately or move to supplemental and eventually go out?

any experience with the same? if yes how long it takes? Any effect to rankings, traffic (which is why I am doing this in the first place)

Please let me know. Thanks in Advance.




 10:14 pm on Dec 20, 2006 (gmt 0)

Yes, that is exactly how robots.txt is to be used.

Just remember that Google isn't the only online source of your indexed pages.


 2:44 am on Dec 21, 2006 (gmt 0)

hi leo,

i think we might want to take a second, step back and try to figure out what you want. generally if you are trying to improve rankings and traffic, link development and not robots.txt is the way to go. also make sure you pages have sufficent unique content.

you mentioned you are doing this because of rankings and traffic. why do you think robots.txt will help you? is it because they are just duplicates? also why dont you want google to index the pages it labeled as supplemental?


 8:29 pm on Dec 22, 2006 (gmt 0)

I want to remove those pages from the index becuase they have little or no content (empty customer reviews etc).

I know it may hurt my page rank on some pages becuase the "real" pages will loose internal link "votes" in google's eyes. But it will atleast cleanup my site index. Based on other threads, it is probably true that Google is penalizing sites becuase of emptpy pages and what not.



 2:01 pm on Dec 23, 2006 (gmt 0)

If you have empty pages and these pages will never gain good content, then it is wise to get rid of these pages.

If you use robots.txt for this you may lose some link popularity. If you use htaccess to do some 301 redirects you can save the link popularity. For me it would depend on how many pages and how many links from other sites are pointing to these pages.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved