Quick removal of Supplememtal URLs that no longer exist?

Mis-spelt and obsolete URLs dominate when using the site: command

[JackR]

Several months ago I first began to notice that around 20% of my pages had gone supplemental. None of these pages have been 'live' for many months - and some existed for a matter of a few hours after new pages were added to the site. The problem is that several URLs were misspelt and these were indexed by Google. They still are - as supplemental results.

All the pages concerned have since been .301 redirected to the home page. Other than waiting many months for them to drop out of the index, is there anything else I can do?

I've considered resurrecting these old URLs and adding content - would that take them out of the Supplemental Index?

[tedster]

If you have fixed the issues involved, then I'd suggest not worrying about the Supplemetal residue. Google will refresh the Supplemental Index on their own schedule, but it is not hurting you. Efforts you take to make the site: results look pretty to your eye may cause you more trouble.

For some reason Google has recently chosen to have Supplemental Results at the top of the site: results for many sites. It can be pretty freaky to see that - not sure why that's happening. But as long as you have taken action to repair the issue, then all should be well.

[g1smd]

Supplemental Results are mainly URLs that are now redirecting, are 404, or are on domains that have since expired.

Google holds on to those URLs for a year before they drop them from the index. That is how they designed it to work.

Make sure that your custom error page contains links to the major section indexes on your site so that the visitor can find their way around.

[Oliver Henniges]

I found the quickest way to get pages off the index is to substitute the file by a dummy-page containing a noindex-meta-tag.

[JackR]

Thank you for the replies.

tedster: Although I'd obviously like the site: results to look like a perfect record of my pages, I'm more concerned about the fact that my main competitors don't have a SINGLE Supplemental page out of hundreds of indexed URLs. Am I right in thinking that this is irrelevant and in no way reflects the final SERPs?

BTW, if I want the site: results to look pretty, I check the 'Pages from the UK' option as that produces a perfect listing. ;)

g1smd: Do you mean add a custom page with text along the lines of 'the page you have requested has now expired, please click here to view our widgets?'

Oliver: I've always been a little wary of using noindex on anything ... just in case. Thanks for the tip though.

[g1smd]

Yes, you want a custom error page with a copy of the main site navigation on it, and a message to say that the requested page has gone....

The same error page is shown for every 404 error that occurs on the site. Make sure that the HTTP status code really is 404 too.

That error page MUST MUST MUST have a meta robots noindex tag within it. The error page must NOT be behind a 301 or a 302 redirect.

[RonnieG]

Supplemental Results are mainly URLs that are now redirecting, are 404, or are on domains that have since expired.

Plus: All interior pages of smaller sites and low PR sites that do not have IBLs to those specific pages to improve their individual PR, regardless of content uniqueness or other factors that in the past would have meant being in the main index.

[g1smd]

Yeah, that is why I said "mainly", but the new type that you note first appeared in just the last month or two and are slowly growing in number.

[steveb]

"Am I right in thinking that this is irrelevant and in no way reflects the final SERPs?"

No, its always a problem, even if a tiny one sometimes, particularly when launching new pages. Supplemental pages reflect on your domains value of pages you launch yourself. For example, constanly launching new pages that are abandoned (say going from 1000 to zero links to them) will make all your new pages have a harder hill to climb to rank. Domains with a lot of cruft, which is one way to explain supplementals, will have a harder time being respected for long tail searches.

Once you get the supplementals though, there isn't much to do besides cleanup and wait. Remove them via the Google remove/hide page tool, redirect them to some unimportant other than the homepage, and move on.

Tidying your site: search results is just part of tidying your domain as a whole.

[RonnieG]

Yeah, that is why I said "mainly", but the new type that you note first appeared in just the last month or two and are slowly growing in number.

Finally, one acknowledgment that this really is happening!

The affected smaller and low PR sites are growing in number as soon as they get crawled, and the impact is becoming more visible every day. In some of the searches I have been doing the last couple of days, relevant results for some of my site's targeted keywords, all localized long tails, are limited to the first couple of pages. Nearly all of the results after that are directories, pure adsense sites and link farms targeting my area and industry. I used to be able to get 10-12 pages deep before I started encountering those.

[g1smd]

Well, acknowledgement of sorts, but with multiple reservations... time and time again I see that it takes 6 to 12 months to figure out what is actually happening, and often when finally figured out it is quite far away from what was initially assumed to be, around the time it was first noticed...

[edited by: g1smd at 12:11 am (utc) on Nov. 13, 2006]

[steveb]

The only thing "new" is Google can now have a main index cache from two weeks ago, and a supplemental cache from six weeks ago; and, Google is discarding pages from the main index far more rapidly than before, leaving only the supplemental page.

I suppose they are more ruthless in choosing to not index domains they view as troubled, but that's just an acceleration of previous actions.