how to kill off dead directories in .htaccess?

stop it already!

[Lorel]

I have directories and their contents that I took off line over 2 years ago but I'm still getting traffic from sites that still have them listed and also search engines because I have a 404/missing.html file set up and it's redirecting all those to my home page.

Is there a way to set up my .htaccess file so the search engines will drop those directories and the files they contained and they will appear as dead links without messing up my good files?

[jdMorgan]

> I have a 404/missing.html file set up and it's redirecting all those to my home page.

That is a problem. A 404 page should not redirect anywhere. You can put a long-time meta-refresh and a text link to your home page on it, but I cannot recommend a server redirect (301 or 302), since that tells the search engine that the 'page' is good, but has a new URL. So they'll keep listing it.

The first step is to use the server headers checker [webmasterworld.com] to be sure that you are returning a 404-Not Found response for these dead pages.

Jim

[BananaFish]

A 301 will tell the spiders the content is permanently move -- put lines something like this in your htaccess file:

RedirectMatch 301 /deaddirctory/.* http://www.example.com/

[edited by: jdMorgan at 2:32 pm (utc) on Sep. 19, 2005]
[edit reason] Example.com [/edit]

[jd01]

I would not recommend redirecting a large number of pages (actually any number) to the home page, but rather handling them according to the HTTP standards, unless they have a large number of inbound links, which you cannot have changed to reflect the changes in your site. If the home page is not truly a replacement for the pages and there are no other pages that should serve as a replacement, the pages should be 'not found' or 'gone'.

If you want the pages removed use a 410.

RewriteRule ^thedirectory/ - [G]

The reason for this is a 404 is 'not found', and served as a default for any page on a site that cannot be accessed at the current time. This could be for a number of reasons, so SEs will *correctly* continue to request the page until they are satisfied it is really removed. A 410 is only served if there is manual intervention -- IOW someone has 'told' the server to serve the 410 status code -- in this case most SEs will drop the pages rapidly, because they know someone has physically removed them for a reason.

Justin

BTW I do not recommend the redirect you are using either -- it may seem nice for visitors, but safer for you to give them a custom error page without the redirect and let them choose the link they would like to follow... This should also help in serving the proper status for a page, as long as you follow the standards when defining them. (Do NOT use a Control Panel for this, hand set them.)

[Lorel]

Hi,

Here is what I currently have in my htaccess file:

ErrorDocument 404 /missing.html

The reason for this is to catch all misspellings and redirect them to the missing page which has links to all the current directories on the site so I don't lose any visitors.

The problem is I have directories that no longer exist (and haven't for over 2 years) that are still listed on websites, search engines that I would like to delete and with the 404 in place that doesn't happen.

I can't use a 301 on them because those pages no longer exist.

jd01 said:

If you want the pages removed use a 410.

RewriteRule ^thedirectory/ - [G]

Do I need to put anything but the directory name in there?

Can you tell me what the [G] stands for?

PS. I didn't mean to say the 404 is redirecting to my home page. the 404 goes to a missing page which has links to other pages including my home page.

thanks for trying to help.

[kevinpate]

> ... what the [G] stands for?

It stands for GONE

[hooter]

How would I do this if I've moved files from this "gone" directory up one level to public document root?

Right now I'm using this in htaccess in root:
RewriteRule ^dead_directory/ http://www.example.com/$1 [R=301,L]

But I've been indexed and linked to files that used to be in dead_directory and now the files are in the domain root.

So using the above, if I have an old link http://www.example.com/dead_directory/this_filename.html
it just gets redirected to
http://www.example.com

How do I get it to redirect to
http://www.example.com/this_filename.html

[jdMorgan]

The problem is that you've got an undefined back-reference, $1, in the substitution URL. Back-references refer to previously-defined parenthesized sub-patterns in the RewriteRule regular-expressions pattern. So your code is doing exactly what you told it to do -- to drop the page name. Corrected, it looks like this:

RewriteRule ^dead_directory/[b](.*)$[/b] http://www.example.com/$1 [R=301,L]

For more information, see the documents cited in our forum charter [webmasterworld.com] and the tutorials in the Apache forum section of the WebmasterWorld library [webmasterworld.com].

Jim

[hooter]

jdMorgan,

All I can say is you are truly a zen-master of this stuff! That worked perfectly :)

Quick follow-up:
To handle requests to the directory alone without trailing slash and have them just redirect to www.example.com I'm using this after the initial rewrite rule and it seems to work fine :)

Redirect 301 /dead_directory http://www.example.com

Do you see any problem with that?

[Dijkgraaf]

Why not use a robots.txt file to and deny the old directories to all bots.
I did that and works fine for stopping bots asking for pages that haven't existed for a while.

[jdMorgan]

There's no need to add another directive to handle these cases. This single directive should take care of all of them:

RewriteRule ^dead_directory(.*)$ http://www.example.com$1 [R=301,L]

Jim