GoDaddy Refuses to Fix Soft 404s for Custom Error Pages

[southernmost]

Despite my pleas, all the way to the President's office, GoDaddy refuses to fix soft 404s when Custom Error page is used.

If you are aren't aware, Google has been insistent that web hosts serve up a true 404 server response code when a page not found error is generated from an invalid url. GoDaddy's shared Windows hosting server (IIS 7.0) has been responding with a 200 HTTP code (instead of the correct 404 HTTP code) if you choose to use the Custom Error page handling. This is known as a "soft 404" and is strongly advised against in Google's webmaster quality guidelines.

FYI, there are three options for error page handling in their shared Windows hosting environment: 1) use the default GoDaddy error page (something that looks horrible and leaves the user unable to navigate to your website), 2) use a Custom page (which would be desirable, since you could make it look like your site and provide useful links for the user), or 3) use your homepage as the error page.

In options 2 or 3, GoDaddy's server will respond with a 200 HTTP error code - meaning that the page exists and was served properly. This is WRONG - as it tells the search engines that the invalid url exists and it is, depending on your choice, your Custom error page or your homepage - which would then be indexed and considered duplicate content...a problem in Google's eyes.

I have begged them to change this. They have admitted that their server does respond with a 200 HTTP code. At first, they thought it was appropriate because they were serving up the Custom error page or the homepage without a problem. But I, at great lengths, explained that this is not the proper use of a 200 HTTP code - because the page being requested does not exist.

After finally understanding this, the office of the president (yes, I reached up the top of the organization) responded that they were aware of the issue, but weren't going to do anything about it - unless, maybe, enough people let them know it was a problem.

[tedster]

It's been this way at GoDaddy ever since I can remember. The situation used to be complicated by the fact that Microsoft training actually used to TEACH IIS admins to use the soft 404 approach to custom error pages.

[netmeg]

I would add that to my ever growing list of reasons not to host with GoDaddy.

[Staffa]

unless, maybe, enough people let them know it was a problem.

Enough people won't let them know. People vote with their feet and go find a better host elsewhere ;o)

[bwnbwn]

another one to add to the list of why not to host there. If you set up a 301 redirect it isn't a real 301 but a 302.

[Content_ed]

I never heard that Google cared about hard versus soft 404's. I thought I was doing everybody (visitors, myself, Google) a favor by doing the IIS redirect to a custom 404 page provided by Google with a search option.

WebMaster Tools reports on Soft 404's with:

"Redirecting to error page"

which I would have thought meant it was cool with them, especially when it's their code on the error page.

[g1smd]

Google reports it so you can go fix it. Those redirects give your site infinite URL space. You don't want that.

There have been several posts on the Google blog about avoiding this method; made popular by the clueless way IIS was designed.

If there is a page that is a close match, content-wise, for the requested URL then by all means redirect (301) to that URL, but otherwise serve a 404 response at the originally requested URL along with a useful page of information for the human visitor.

[southernmost]

Content_ed: On December 6 of this year, Google released an article called "Tips for hosting providers and webmasters" on their Official Google Webmaster Central Blog, in which they wrote:
"Soft error pages. Some hosting providers show error pages using an HTTP 200 status code (meaning “Success”) instead of an HTTP error status code. For example, a “Page not found” error page could return HTTP 200 instead of 404, making it a soft 404 page; or a “Website temporarily unavailable” message might return a 200 instead of correctly returning a 503 HTTP status code. We try hard to detect soft error pages, but when our algorithms fail to detect a web host’s soft error pages, these pages may get indexed with the error content. This may cause ranking or cross-domain URL selection issues.
It’s easy to check the status code returned: simply check the HTTP headers the server returns using any one of a number of tools, such as Fetch as Googlebot. If an error page is returning HTTP 200, change the configuration to return the correct HTTP error status code. Also, keep an eye out for soft 404 reports in Webmaster Tools, on the Crawl errors page in the Diagnostics section."

and in another Google article, "Soft 404 Errors" in the Webmaster Tools Help section, Google states:

"Returning a code other than 404 or 410 for a non-existent page (or redirecting users to another page, such as the homepage, instead of returning a 404) can be problematic. Firstly, it tells search engines that there’s a real page at that URL. As a result, that URL may be crawled and its content indexed. Because of the time Googlebot spends on non-existent pages, your unique URLs may not be discovered as quickly or visited as frequently and your site’s crawl coverage may be impacted (also, you probably don’t want your site to rank well for the search query [File not found])"

[robzilla]

If they can't even take care of the basics of web hosting, I'd take my money elsewhere. I haven't seen this "default GoDaddy error page", but I'm familiar with Godaddy's approach to marketing their services. I can imagine this is just another way of improving their own bottom line.

[Content_ed]

Interesting. I just stuffed the pages that were returning soft 404's into robots.txt, There weren't many of them in the history, half dozen for one site and a dozen for the other. Of course, there weren't any on my non-Pandalized site:-)

[deadsea]

Despite what Google's documentation says, Googlebot is very good at detecting soft 404s and doing the appropriate thing. I would suggest using the custom error page and making the title "404 Not Found" and making a prominent H1 be "404 Not Found". With these obvious signals, Googlebot WILL detect the soft error page and treat it appropriately despite the dead-headedness of your hosting.

[dstiles]

If you're using a custom 404 page can't you alter the headers?

Works for me on IIS, where I make decisions as to whether to redirect (eg was .html but is now same page .asp), whether to send a 404 or, in extreme cases, a 403, very useful to deflect trojan-injectors and their ilk.

[incrediBILL]

Google has been insistent that web hosts serve up a true 404 server response code when a page not found error is generated from an invalid url.

Google isn't THAT insistent or Google wouldn't be putting so much effort into detecting soft 404s in the first place.

Heck, I detect soft 404s when link checking, it's something that's so rampant if you don't do it the link checkers used by most are quite ineffective these days.

BTW, domain parks are the worst offenders, they don't want anyone to know the site has been moved, repurposed, and stuffed full of ads so all page requests are 200 OK.

[Swanson]

This is a mis-direction. Soft 404s are detected fine by Google and as Godaddy are one of the largest hosts in the world then you should expect they have "had a chat".

I host thousands of domains with Godaddy - soft 404 has never been a problem.

To the OP - has this handling caused a drop in traffic or any other reason that you should be worried about - or are you worried about breaking "the rules"?

I often find if there is no problem at the moment then life is too short to worry or change things - because things can change after by google no matter what you do. If it's not broken then don't fix it.

[g1smd]

It is broken and Google warns that it might cause problems leaving it like that.

That's why they go to the trouble of warning you in WMT that there's a problem.

[tangor]

The fact that G has been taking time recently to point out the errors, and difficulties of infinite URL space (soft-404s as 200, etc.) it seems possible that at some time in the near future there will a new switch added to the G algo: soft-404 fail, demote site-50, we warned ya!

[enigma1]

It is broken and Google warns that it might cause problems leaving it like that.

No, it just means you don't know how to customize an error page.

In options 2 or 3, GoDaddy's server will respond with a 200 HTTP error code - meaning that the page exists and was served properly.

Right, therefore it leaves you with option-1 at the moment. So either change the code of your site to support them or use the default error page of the host.

And providing the same 404 page for invalid requests doesn't help visitors.

I don't know how much they'll demote a site for soft 404s I believe is subject to problems with the site's internal link structure.

[southernmost]

To the OP - has this handling caused a drop in traffic or any other reason that you should be worried about - or are you worried about breaking "the rules"?

Yes, there has been a drop in traffic - a la Panda.

So, in an effort to find any little thing that could contribute to the problem, I've been trying to solve the soft 404 issue (which Google Webmaster Tools has NOT been warning me about, but has been alerting about a list of 404s).

[southernmost]

And providing the same 404 page for invalid requests doesn't help visitors.

Not sure what you meant here, but providing the same 404 page for invalid requests DOES help visitors, since it gives them links to popular content and keeps them on your website.

[rlange]

southernmost wrote:
After finally understanding this, the office of the president (yes, I reached up the top of the organization) responded that they were aware of the issue, but weren't going to do anything about it - unless, maybe, enough people let them know it was a problem.

I do not and have not used GoDaddy for hosting, but this response still angers me.

This issue is such a fundamental piece of web hosting and they're doing it wrong. Worse than that, they acknowledge that they're doing it wrong, but won't do it right unless an unspecified number of people complain about it.

GoDaddy clearly doesn't take the services they provide very seriously.

The proper customer response to this is to cancel your account and clearly cite this issue as the reason. Don't just complain and then continue to pay them, because they'll have no incentive to improve.

--
Ryan

[bwnbwn]

Go daddy offers a good beginner hosting service. Time to move away and move up to a better host.

[netmeg]

No, it just means you don't know how to customize an error page.

For anyone unfamiliar - go back and look at any or all of g1smd's posts over the years, and make your own conclusions as to the extent of his knowledge on this subject.

sheesh, no way I was gonna let THAT stand

[enigma1]

@netmeg did you even read the OP? Probably not.

GoDaddy's shared Windows hosting server

Shared server and custom error page. Same server multiple accounts. If you want to send a specific header out like 404 do it for your site and in your code with a custom page. Someone else wants to handle it differently.

Many shared servers work like this, seems first time you heard about it not everyone wants to do it that way. If you need more control rent a dedicated or virtual box.

[snickles121]

I use Godaddy with a custom 404 page. When I check it under google webmaster tools under googlebot fetch it returns:

HTTP/1.1 404 Not Found
Date: Thu, 15 Dec 2011 19:42:33 GMT
Server: Apache
Accept-Ranges: bytes
Content-Length: 10262
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

so it must be ok then or it would return a 200 code.

[southernmost]

I use Godaddy with a custom 404 page. When I check it under google webmaster tools under googlebot fetch it returns:

HTTP/1.1 404 Not Found

snickles121:

How did you achieve this? web.config? On-page server response code? What type of page is it? .html? .aspx? etc

I definitely get a 200 response code when using a Custom error page. My site is a static site with .html page extensions (including the error page).

And GoDaddy has confirmed that they return a 200 code with Custom error pages.

[enigma1]

The proper way is on a shared server, when you override the error page, the server should not output a specific response header.

You don't know what each site wants to do for an error how it's going to handle it and the programming language used. For PHP it implements a header override option with recent versions. ASP has an AddHeader method that behaves differently.

The AddHeader method adds a new HTML header and value to the response sent to the client. It does not replace an existing header of the same name. After a header has been added, it cannot be removed.

reference:
[msdn.microsoft.com...]
So if the server outputs 404 always including a custom page, your code may not be able to alter it.

[snickles121]

Southernmost....

Go into your hosting control panel, then click on the green tab at the top that says settings. Next click on 404 error options in the drop down menu. Then use the custom 404 error page and enter the page you want to use for your error page. Then hit continue and then submit it.

Thats at least how I did it with them to get it to work right. Let me know.

P.S. I use a standard html page with my regular template. No special codes on the page just the standard.

[snickles121]

I just figured out why you got confused. When you do a googlebot fetch for the custom 404 page itself it does return a 200 code, but if you do it for a page that does not exist like I did the first time it returns the proper 404 code.

If your worried about that one page getting indexed just use the no index, no follow header in that error page. I been using this custom page for a year and never had a problem with it yet.

P.S. I just wanted to add that the only time this would be a problem is if you got a 200 code every time you type in a page that didnt exist. But, as long as it shows up a 404 code for the pages that don't exist and still shows your custom error page, you will be OK.

Hope this makes sense..

[Swanson]

snickles121, yes - that is it and why I got confused about the issue.

I was also mis-using the terminology - I was using "soft" as a replacement for "custom". I didn't realise the definition of a soft 404 returned a 200 - which to me is not a 404 at all. You learn something everyday!

A custom 404 should never return a 200 HTTP response code - yes you get all the problems that g1smd mentioned (he is the expert on these issues I have seen over the past few years).

However you shouldn't get a 200 response code on the invalid URLs - yes you will if you test headers on the actual 404 custom page as it is valid.

[southernmost]

Southernmost....

Go into your hosting control panel, then click on the green tab at the top that says settings. Next click on 404 error options in the drop down menu. Then use the custom 404 error page and enter the page you want to use for your error page. Then hit continue and then submit it.

Thats at least how I did it with them to get it to work right. Let me know.

P.S. I use a standard html page with my regular template. No special codes on the page just the standard.

Snickles: this is exactly how I set up my Custom error page.

I just figured out why you got confused. When you do a googlebot fetch for the custom 404 page itself it does return a 200 code, but if you do it for a page that does not exist like I did the first time it returns the proper 404 code.

I'm getting a 200 code when an invalid (non existent) url is requested. I understand that if I directly request my error page, then I get a 200 code and that is proper. But what I see happening is a 200 code when an invalid url is requested. It should be a 404 being served up with the Custom error page.

Here is the "fetch as googlebot" result from Webmaster tools:

HTTP/1.1 200 OK
Content-Type: text/html
Last-Modified: Wed, 14 Dec 2011 15:58:30 GMT
Accept-Ranges: bytes
ETag: "beb4853979bacc1:0"
Server: Microsoft-IIS/7.0
X-Powered-By: ASP.NET
Date: Fri, 16 Dec 2011 01:45:02 GMT
Content-Length: 22711

As you can see, I'm getting a 200. The url I requested was something like: www.mywebsite.com/something-random-and-nonexistent