Forum Moderators: phranque
basically i have a bunch of dynamically generated pages that have been indexed over the years that when they show up as a search result ultimately end up going to the same page with the same content. i'm not sure how bad that effects my rankings in the search engines but either way these urls are no longer valid and i want to avoid adding 10,000 entries into my robot.txt file.
Every now and then I need to nudge Google into (re)heeding my robots.txt file(s) and request removal of specific directories and docs, particularly .txt files where "noindex,nofollow" kinds of tags don't apply.
I use their URL Remover [services.google.com], offering a number of Options you might find helpful, specifically: "Remove pages, subdirectories or images using a robots.txt file." (You'll need to create an account to access the features.)
Caveat: Be sure your Options instructions are correct because --
"Please keep in mind that submitting via the automatic URL removal system will cause a temporary, six months, removal of your site from the Google index..."
(I haven't found that to be a problem vis-a-vis my robots.txt-related 'updates' but YMMV.)