301 Redirects into a 404 page

getting old urls out of search indexing

[njm5785]

We have old urls that are being 301 redirected to our new url. For some reason a couple of the old urls are still showing up in search indexing.

Here is what is happening:

oldurl.com/directory/ -301-> newurl.com/directory/ (404 returned)

Our new website does not have some of the directories that the old website did so it returns the 404 page.

What is the best way to get those old urls taken out of the search engines?

[Receptional Andy]

Hi and welcome [webmasterworld.com] njm5785 :)

You need to return a 404 (not found) or 410 (gone) for oldurl.com/directory/. Why are you redirecting these URLs if they don't exist?

[njm5785]

It is redirect automatically because we are using a rewrite rule in our server config file to take anything from that old url and send it to the new one and also keep the file path intact.

Would it be best to write conditions in our rewrite rule that told it to skip those specific pages?

[g1smd]

Ideally, the old URL should directly return 404. That's the best way to fix the problem.

If there is a 301 redirect from one URL to another, and the final URL does return a 404, then Google will eventually "get the message".
It's no big deal - as long as there is not a 302 redirect in there.

If you add an exclusion for the redirect, that will fix the problem quicker, but will be harder to maintain in the long run.

[edited by: g1smd at 10:10 pm (utc) on April 9, 2008]

[Receptional Andy]

A good enough reason ;)

More questions, I'm afraid. How many URLs are involved? Is there a pattern shared by those URLs that still exist or those that don't? Why are these URLs unavailable?

Added: To second g1smd, it's unlikely to cause any search engine problems, unless on a large scale. But URL housekeeping is good ;)

[edited by: Receptional_Andy at 10:11 pm (utc) on April 9, 2008]

[njm5785]

g1smd - there is not 302 in there just a 301 straight to a 404.

Andy - it is only 5 urls, that i know of, and they have nothing in common except the domain. With that few it is not too hard. They are unavailable because we no longer have those topics.

The reason we are trying to figure this out is because we are working with an SEO company and they say this, along with a few other little things, is why we are not getting good rankings in the search engines.

[Receptional Andy]

The reason we are trying to figure this out is because we are working with an SEO company and they say this, along with a few other little things, is why we are not getting good rankings in the search engines.

Hopefully they have given you some idea of what the likely results/impact of this (and other) changes might be. I think it's worth fixing, but frankly, it isn't going to improve your search engine performance if there are only five URLs.

I wouldn't judge a company based on a snippet in a forum post, but I'd expect that if your search engine performance is poor, technical fixes would be a small part of the SEO process (unless there are large-scale problems).

[njm5785]

we have other things that we need to fix too, but this is just the area i was working on right now. Since they said it was part of the problem i figure if we get everything fixed then that will help overall.

Thanks for the help.

[g1smd]

The first thing that I nearly always work on, is fixing the URL footprint that a site exposes for indexing.

After that it's code, internal linking, tags, content, and all the other good stuff.