Why don't the SEs remove my old pages from their indexes?

And why won't they add my new ones!?

[nonprof webguy]

In August 2002, the site I manage re-launched with a completely new structure, lots of new content, and an entirely different set of page URLs.

I knew at the time that changing the page URLs was going to cause problems for our visitors because there are thousands of links on the web to our old site's URLs, and hundreds of pages listed in search engines that are for those old URLs. Our 404 page instantly became the "most popular" page once we launched the new site.

I couldn't worry too much about all the links out there, but I figured the search engines would eventually catch on, remove the old pages from their indexes and add the new ones. After all, the major SE crawlers visit us with frequency.

But now, here it is nine months later and Google and most other search engines continue to list a few hundred nonexistent pages that were once on our site. Meanwhile, they don't seem to have discovered a few hundred pages of our 'new' site.

Looking at my logs I can that the bots aren't doing a deep crawl, but simply checking for pages already in their indexes, or checking links they found on other web sites.

I've set up my 404 page to detect what page the visitor is seeking and redirect them with an appropriate status code and url. For the humans, this means most don't actually see a 404 page, but get served the page they were seeking at its new url. Even though I send a 403, the bots update the title and description in their indexes, only, but not the url!

It's finally getting my a little exasperated. There are pages that haven't been on our site since 1995 that are still listed in some major search engines! What else can I do? I'm at a nonprofit organization with limited time and resources. What do you suggest I do to attack this problem that will get the most bang for the buck?

[Brett_Tabke]

hmm. I've heard of situations like this before, but not quite this bad.

The situation is actually exploitable from an seo standpoint. I've got a client that has moved urls four times since 1997 and kept control over the four domains in the process. Inktomi still thinks they are four separate websites (all three old urls still feed the main site quality traffic off server side redirects).

Generally, a good 301 to the new page will do the trick.

If that doesn't, a meta refreshed "stopper" page can help. Put "Page Moved" in the title, and the traditional "click here to continue" link on the page. As the text of the link, put the new link address.

[nonprof webguy]

Good point, Brett, and thanks for responding. The bright side of the matter is that there are still several hundred pages of ours in Google, and even though they're wrong, about 80% of visitors from SEs get automatically forwarded to the correct page by my error page's server-side script.

Correction:

Even though I send a 403, the bots update the title and description in their indexes, only, but not the url!

I meant that I send a 301, moved permanently, not 403 forbidden... Sorry!