Error Handling

delivering 404 header to SE's when redirecting

[sssweb]

I've been told that error handling for my site is negatively affecting search engine spidering.

I have the following in my root .htaccess file:

ErrorDocument 404 /error-nav.php?error=404

and error-nav.php redirects either to the correct page or /not-found.htm

Can someone please tell me exactly how to send a proper 404 header to SE's without affecting how user's are redirected?

Is this done via .htaccess, my error-nav.php file, or my not-found.htm page? And what code do I add and where?

[phranque]

your .htaccess looks ok.
i assume error-nav.php is doing a 301.
if it redirects to the correct page, that page should return a 200 OK status.
the not-found.htm should be providing the 404 error, either in the http headers or the http status line.

writing the HTTP status is simply a matter of starting the http document with something like:

HTTP/1.1 404 Not Found

or the following in the HTTP header portion of the document:

Status: 404 Not Found

the browser/spider sees each response and makes a new http request if appropriate.

[sssweb]

I changed not-found.htm to not-found.php and added the following at the start:

<?php
header("HTTP/1.0 404 Not Found");
?>

That works fine for the header on that page. The problem is that the first header the SE gets (and therefore the critical one in determining the status of the URL it's spidering) is a 302, which is the 'temporary redirect' from error-nav.php

That code is:

header( "Location: [".$_SERVER['SERVER_NAME']."...] );

I've tried adding:

header("HTTP/1.0 404 Not Found");

and/or

header("Status: 404 Not Found");

before and after that line, but it either has no effect on the header or, if it does output the correct 404 header, the script doesn't redirect users to not-found.php. They simply get a generic 'not found' page. I've also tried using the various flags that go with the header() function (see [ch2.php.net...] ), but still get the same results.

[jdMorgan]

The problem is that you shouldn't be *redirecting* until you know whether the document exists or not. As soon as you do a redirect, that is the end of the current HTTP transaction, and the client's request context is lost. At that point, it is 'too late' to indicate a 404-Not Found.

Speaking in the most general of terms, try *including* the error-handling code inside /error-nav.php instead of redirecting to it.

There are other solutions, but most are more complex.

Jim

[edited by: jdMorgan at 1:38 am (utc) on Dec. 15, 2006]

[sssweb]

The page DOESN'T exist; that's why I need to send a 404 header before or with the re-direct. I've tried, as you suggest, including it in my error-nav.php code, but I'm not versed enough in this to get it to work.

The redirect code in that file is:

header( "Location: [".$_SERVER['SERVER_NAME']."...] );

I've tried the following variations (and other similar ones), without success:

header("HTTP/1.0 404 Not Found"); 
header("Status: 404 Not Found"); 
header( "Location: http://".$_SERVER['SERVER_NAME']."/not-found.php" ); 

header( "Location: http://".$_SERVER['SERVER_NAME']."/not-found.php" ); 
header("HTTP/1.0 404 Not Found"); 
header("Status: 404 Not Found"); 

header( "Location: http://".$_SERVER['SERVER_NAME']."/not-found.php", , 404 ); 

header( "Location: http://".$_SERVER['SERVER_NAME']."/not-found.php", false, 404 ); 

(see [ch2.php.net...] for explanation of last two)

It seems to me this is a common issue. Almost every site has a custom error page, that the user is brought to via a redirect. How do webmasters deal with outputting the correct 404 header to SE's?

[That_Guy]

Try this:

Use this in error-nav.php:

header("HTTP/1.0 301 Moved Permanently");
header("Status: 301 Moved Permanently");
header( "Location: [".$_SERVER['SERVER_NAME']."...] );

Keep this in not-found.php:
<?php
header("HTTP/1.0 404 not found");
?>

Let me know if it works...

Edit (after second cup of coffee):

I believe Apache will send a redirect when using an absolute url, even if it is on the same server. Try:
header( "Location: "/not-found.php" );
in your error-nav.php, this should allow the original 404 header produced by your .htaccess (which is correct as above) to pass through.

[edited by: That_Guy at 3:02 pm (utc) on Dec. 22, 2006]