Welcome to WebmasterWorld Guest from 220.127.116.11
We setup a vb sub that will check for these parameters in the URL. If these parameters are MISSING it will display <META NAME="Robots" CONTENT="noindex"> If the parameters are INCLUDED in the url it will display <META NAME="Robots" CONTENT="index, follow">
Will this stop ONLY the unwanted url from being listed in google, or will this cause the entire page file to be dropped?
After reading many posts by g1smd, I think this setup will work, but this seems important enough to ask before implementation.
If they are Supplemental, they will take a lot longer to disappear, but Google will get rid of them eventually. It may be months. It might be a year.
Only the URLs that serve a noindex tag will be dropped. Others will remain.
From my experience that will work quiet well. Anyway - if you are able to find another solution it might be better. Googlebot has to read all of your documents first. And then they need to be processed. That takes some time...
If some of your documents are already marked as supplemental it may take a very long time to remove them.
Do you have lots of urls?
However, I've noticed that Googlebot is crawling the parametered URLs on a daily basis. The pages aren't indexed, as they were done this way from new, but I'm surprised at the regular ongoing crawling of pages with 'noindex'.
MSN, incidentally, has completely ignored the noindex and has indexed all the parametered URLs.
[edited by: Patrick_Taylor at 10:51 am (utc) on Sep. 28, 2006]
It must work that way, otherwise changes that you make will never be picked up.
What they crawl is a larger number of URLs than what they index content for. What they index is a larger number of URLs than they show in the search results.