404 handling

[gulliver]

My current setup serves custom 404 pages - but not if an url with suffix .php or .cgi is entered. (There may also be other instances.)

How do I sort this?
My .htaccess reads:

ErrorDocument 404 /error/

UPDATE...

I fixed the issue by adding to the .htaccess file. It now reads:

RewriteEngine On
RewriteCond %{REQUEST_FILENAME}!-f
RewriteCond %{REQUEST_FILENAME}!-d
RewriteRule (.*) /error/

ErrorDocument 404 /error/

[jdMorgan]

You might want to check your server error response [webmasterworld.com] for the missing-cgi-file case. Since you are rewriting directly to your error page, I suspect your server will return a 200-OK response for missing cgi pages. This can led to serious problems with search engine listings.

A quick fix would be to rewrite the missing files to a non-cgi file that you *know* does not exist:

ErrorDocument 404 /error/
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# Rewrite missing cgi files to known non-existent html page to force proper 404 error handling
RewriteRule \.(php¦cgi)$ /force_404_error.html [L]

Replace the broken pipe "¦" character above with a solid pipe before use.

This problem can be caused on Apache versions below 2.0 if the module load list is not correctly ordered. Under Apache 1.x, modules are processed in the reverse order loaded. This can cause problems like improper error handling and failing to run mod_rewrite for cgi script requests. If you have control of your server configuration, this would be worth looking into.

Jim

[gulliver]

Thanks, J.

Now I'm totally confused. ;-)

I did the original rewrite after looking at the host (1and1) support pages.

I used the script you suggested:

Entering a non-existant directory name or text with a .htm/.html suffix produces a slightly different message than something with a .cgi/.php...

HTTP/1.1 404 Not Found
Date: Sun, 30 Jan 2005 18:07:23 GMT
Server: Apache/1.3.29 (Unix)
X-Powered-By: PHP/4.3.10
Connection: close
Content-Type: text/html

HTTP/1.1 404
Date: Sun, 30 Jan 2005 18:08:01 GMT
Server: Apache/1.3.29 (Unix)
Connection: close
Content-Type: text/html

I have no idea what this means.

Advice appreciated. Please and thanks.

[jdMorgan]

As long as you're getting a 404 in the first line, you'll be OK.

But it's very bad news if you get a 200-OK in response to a request for a non-existent page. This can make your site appear to have an infinite URL-space, and search engines will limit their depth of spidering on such sites, so as not to spend 'forever' trying to index the site. This only happens if a search engine finds a bad link to your site, but it does happen.

Jim

[gulliver]

Thanks. Appreciated. ;-)