We are trying to stop any duplicate content caused when page parameter is missing from the URL.
We setup a vb sub that will check for these parameters in the URL. If these parameters are MISSING it will display <META NAME="Robots" CONTENT="noindex"> If the parameters are INCLUDED in the url it will display <META NAME="Robots" CONTENT="index, follow">
Will this stop ONLY the unwanted url from being listed in google, or will this cause the entire page file to be dropped?
After reading many posts by g1smd, I think this setup will work, but this seems important enough to ask before implementation.
Msg#: 3099624 posted 10:24 am on Sep 28, 2006 (gmt 0)
We are using the "noindex, no follow" tag on may things. Things will eventually delist. The keyword is "eventually". Not sure how long it actually takes. (Weeks? Months?) -- I would guess that it will be months.
Msg#: 3099624 posted 10:37 am on Sep 28, 2006 (gmt 0)
From my experience that will work quiet well. Anyway - if you are able to find another solution it might be better. Googlebot has to read all of your documents first. And then they need to be processed. That takes some time...
If some of your documents are already marked as supplemental it may take a very long time to remove them.
Msg#: 3099624 posted 10:48 am on Sep 28, 2006 (gmt 0)
I'm doing pretty much the same thing, except that I'm trying to stop any duplicate content caused when a parameter is present in the URL. I want only the plain URL to be indexed and my pages throw up the noindex meta when there's a parameter.
However, I've noticed that Googlebot is crawling the parametered URLs on a daily basis. The pages aren't indexed, as they were done this way from new, but I'm surprised at the regular ongoing crawling of pages with 'noindex'.
MSN, incidentally, has completely ignored the noindex and has indexed all the parametered URLs.
[edited by: Patrick_Taylor at 10:51 am (utc) on Sep. 28, 2006]