Search engines and '401 Authorization Required'

[Asia_Expat]

The well known forum software IPB's version three is presently in release candidate and I've been working hard to get it modified for my own website. One of the things that IPB never seems to get right are the staturs codes on error pages, which return '200 OK' out of the box. I've managed to figure out a fix for this in the core files and can now produce whatever error I choose on error pages e.e. topic not found, member only forums etc etc...

I used to use '410 Gone' on all error pages but I've been rethinking this, because it produces the code on links that are still there to guests and I guess it's not a good isea to have links everywhere that produce a 410 header... so I've now decided to use '401 Authorization Required' on all error pages on my forum.

My question is, how are the search engines likely to deal with those pages, and the site as a whole?

[encyclo]

410 Gone specifically means that the resource did exist once, but has since been permanently deleted. Unless this is an accurate description of the situation, you should not be using this response code.

401 Authorization Required is usually in conjunction with basic authentication, and a browser-generated username/password dialog is shown. I've not ever seen/tested using a 401 outside this specific context - I think you will be better leaving a 200 OK response code and showing the in-page login dialog as usual.

If you return an error because a topic is not found, then you should return a 404 Not Found response code and not a 410 - it's the most logical response code, and it is the standard response code for non-existent pages that is fully recognized by search engines.

In most cases, major search engines will ignore or de-index pages with 403, 404 or 410 response codes. Some smaller search engines may not recognize 410 at all, and for the others a 410 is usually treated the same as 404.

[Asia_Expat]

Thanks.
I've actually reconsidered and changed it to a 404. Unfortunately, I've not figured out a way to tailor the error for specific error pages, so forums hidden from guests etc are producing a 404 response... so there are now 404 links on every page.

I figured the risk of having 404 links everywhere is the lesser of the two evils, the other being non existent /deleted topic pages resolving a 200 OK.

[edited by: Asia_Expat at 3:02 pm (utc) on June 14, 2009]