Code 200 On a Custom 404 Page.

Is this really happening?

[mattglet]

It might be that I'm interpreting this incorrectly, but it looks to me that Freshbot is getting a code 200 from my custom 404 page. Here's a snippet from my log, with the status code at the end:

GET /404.asp 404;http://www.widgets.com/privacy.htm 200

Now, is this b/c Freshbot has actually indexed this, and just checking to see if it still exists? I am confused. Thanks in advance.

-Matt

[bcc1234]

Request your page through a telnet client and check the code that is returned by the server.

Googlebot sees the same thing.

[mattglet]

But why would it get indexed in the first place? Wouldn't the only way to get to a 404 page, is via a 404 return code?

[WebGuerrilla]

Your server is not figured correctly. When you make a request for a page that doesn't exist, your server is sending a 302 Object moved instead of a 404.

The location it gives is your 404.asp page. When a spider gets a 302, it will request that page. If that page is working, it will get a 200 and index it.

[mattglet]

How do I check to see if this is the case? (I do not have telnet access to my site)

[WebGuerrilla]

You can use the WebmasterWorld Header Tool [webmasterworld.com]

[bateman_ap]

It's actually a good way to make your dynamic pages look static.

[mattglet]

Uh oh... Something's not right. If I use the header tool to check a bogus page that has a .asp extention, I get a 302 error. If i try with any other extention, I get a 200. This is honestly happening.

[mattglet]

Also, how do I get my hosting company to fix this?

[jatar_k]

You will have to explain it to them and make sure you get someone who understands what a header code is.

If they do understand it shouldn't be too difficult to explain to them.

[mattglet]

re: webguerilla's post above about the 302 status...

should i be getting a 302, or a 404 when using custom 404 pages (assuming all settings are correct)?

[jatar_k]

It depends on what you want to happen.

A regular 404 page should give a 404. This will handle the situation when someone requests something that doesn't and exist and you have no reason to handle the possibilty of it ever existing.

302/301 should be used when existing pages are renamed or moved. Meaning you want to handle it differently, anyone who requests that page/resource need to be sent to the new location.

302=moved temporarily
301=moved permanently

[jomaxx]

Even a custom 404 page should return status 404.

[jtl]

I Think the reason for the results you are seeing is because the “message type” is set to “URL” in the “error mapping properties” on a windows NT server, under custom errors tab, in site properties, I to suffer the same problem because my custom 404 error page has asp code in it, so it needs to be executed, if your custom 404 error page does not have executable code on it like asp, then change the file extension to “htm” if it is not already then get your host to change the message type to “file” instead of URL this should then return a 404 in the header.

[mattglet]

jtl was right on the money. although, it creates a new problem.

b/c my 404 page has asp code in it, when i change the message type to file, it processes no server side code. is there a way to get around this problem? with a message type of "file" google treats this as a 302, when it really needs to be a 404...

[JuDDer]

You should be able to keep your ASP page and in IIS, have it set as 'URL' and not 'File'.

In your ASP page, something like this should return the 404 header that you are looking for:

<%
Response.Buffer = True
Response.Clear
Response.Status = "404 Not Found"
%>

[mattglet]

that's a negative. still returning 302, not 404.

[JuDDer]

Very strange!
Is that code right at the top of your page?
Have you tried this error page anywhere else such as on a local server?

[mattglet]

<%option explicit%>
<%
Response.Buffer = true
Response.Clear
Response.Status = "404 Not Found"
%>

that's the top of my 404 page. i have not tried it on any other server than my live server (which i do not host, btw).

[JuDDer]

I'm pretty sure that code should be making the server return a 404 header.

I dont know if it makes any difference, but you may want to check with your host that in the url field when setting your custom error, they have it set as /yourfolder/errorpage.asp rather than yourdomain.com/yourfolder/errorpage.asp.

[zlb_17]

i'm confused about something.
on one of my clients' site, i have set up the htaccess to handle the 404 error by taking the user to the site-map page. this is because many people would link to their site through old and obsolete urls. this way, the visitors can still find what they need even if they arrive on a 404 error, instead of taking them straight to the index page or to keep them on a "page not found" 404 page.
would google understand that there IS an inbound link? even though the page it is linked to doesn't really exist anymore, but the site still does? would google index this site-map page as if it was the page originally called? then i would have many duplicates, and the site could be seen spammy? in case of a 404 referrer, the header of the map page says "you were looking for such and such url which does not exist anymore, blah blah", so the pages are not exactly the same, but 99% is, so it could be seen as spammy if the page is indexed several times and looks like many duplications of the same page. or would it see that it's a 404 and/or a 302 event, and therefore index it only once, with its true /map.php url, and that's it? should i be worried? (actually, i'm staring to be, right now...)
thanks.