Welcome to WebmasterWorld Guest from 54.146.174.220

Forum Moderators: phranque

Check Server Headers

New Year's Resolution for all...

   
12:30 am on Jan 3, 2007 (gmt 0)

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Please, if you do one thing in the next 24 hours, check your server headers and be sure that the proper http status codes are being returned. As an added twist, find a tool that does recursive lookups. If all you are seeing is the destination response, you may be missing out on some important responses along the way to that final destination response.

I just caught a 302 loop on a particular domain. Using most server header tools, I can only see the final response which is a 302. But, it doesn't tell me that there is a loop occurring. :(

Please, double check and triple check your rewrite directives and make sure that proper server responses are being returned for ALL URI queries.

For those of you with a custom 404 in place, be sure to check the server header response on that one too. If you are returning a 200 status instead of a 404, there may be problems.

11:36 am on Jan 3, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks pageoneresults, just the topic I wanted to ask about :)

We use a cms which generates a custom 404 page if a page doesn't exist. This thankfully doesn't happen too often as our cms practically eliminates broken links, but this page does appear if a user types in a wrong url for example. I guess it would also appear if someone had bookmarked a page which subsequently disappeared, but at least 90% of our pages are probably permanent.

I was trundling around on ww the other day and came upon the header checking tool, and I tested our custom 404 page, and found that it was giving 200 OK status instead of 404. I asked our cms provider about this and they said that was the correct behaviour, and it shouldn't generate a 404.

Is this correct? Everything I've read here seems to suggest otherwise. How much of an issue is this and what problems could it cause? The site in question is a public sector one and we seem to do OK in Google, and bearing in mind that we don't generally have broken links or a fast turnover of pages, should we be concerned at all?

3:04 pm on Jan 3, 2007 (gmt 0)

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Is this correct?

If you test a URI that does not exist for your site, what is the status returned in the header? A 404 or a 200? If a 200 is returned for a URI that does not exist on your site, then it is not set up properly and may cause problems.

P.S. I'd also check the www vs. non-www and be sure one is 301'd to the other. I see some sites using a 302 when they should probably be using a 301. ;)

4:31 pm on Jan 3, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



re: www vs non

If in the browser URL address I enter http://www.example.co.uk/ that is fine.

If I enter http://example.co.uk/ I get served that page with status 200.

I guess that is not correct as surely it should redirect to www.widgets.co.uk?

In httpd.conf I have this:


RewriteCond %{HTTP_HOST} ^example\.co\.uk
RewriteRule ^/(.*)$ http://www.example.co.uk/$1 [R=permanent,L]

So it looks as if something is not working?

[edited by: encyclo at 10:44 pm (utc) on Jan. 4, 2007]
[edit reason] switched to example.co.uk [/edit]

4:49 pm on Jan 3, 2007 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



HelenDev,

> I asked our cms provider about this and they said that was the correct behaviour, and it shouldn't generate a 404.

Well, they are wrong. As PR1 clearly states, any URL that does not resolve to an existing page should return a 404 status -- end of story, no wiggle-room, period, full stop. Otherwise, there is nothing to tell search engines not to index and list that URL in their search results. You risk you site's theme being diluted, and over time, shifting from its original subject to the theme, "This page not found".


Frank_Rizzo,

> surely it should redirect to www.example.co.uk

Yes it should. Is there a "RewriteEngine on" directive somewhere above that rule, as required to enable rewriting?

Jim

[edited by: encyclo at 10:44 pm (utc) on Jan. 4, 2007]

6:24 pm on Jan 3, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Here's the full section dealing with that stuff:


<Directory "/home/widgets/public_html">
Options FollowSymLinks

<Files ~ "^.*$">
order allow,deny
allow from all
deny from 198.133.178.0/23 #central new mexico college
#many other's listed here - removed for display purposes
</Files>

RewriteEngine on
RewriteCond %{HTTP_HOST} ^example\.co\.uk
RewriteRule ^/(.*)$ http://www.example.co.uk/$1 [R=permanent,L]
RewriteCond %{SCRIPT_FILENAME} ([^/]+)\.wmv$
RewriteRule ^.*$ http://www.example.co.uk/misc/misc.html [R=permanent,L]
</Directory>

The script filename / .wmv section is a workaround for a self loading video file. Is this causing the redirect not to work?

[edited by: encyclo at 10:44 pm (utc) on Jan. 4, 2007]

9:35 pm on Jan 4, 2007 (gmt 0)

5+ Year Member



Frank, the slash at the beginning of your rewrite rule is not wanted. It should be like this:


RewriteRule ^(.*)$ http://www.example.co.uk/$1 [L,R=301]

[edited by: encyclo at 10:45 pm (utc) on Jan. 4, 2007]

12:33 pm on Jan 8, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks for the clarification, that's what I thought :)

Presumably you can still have custom error pages and send a 404 header? Our CMS provider seems to be suggesting that sending a 404 header would mean losing the friendly error pages...

1:04 pm on Jan 8, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



HelenDev:
Our CMS provider seems to be suggesting that sending a 404 header would mean losing the friendly error pages...

Not fully accurate.

MSIE has "friendly" (sic) 404 pages.

Translation: (almost) no-one using MSIE will see your own 404 pages. They will instead see Microsoft's idea of a friendly 404 page.

1:09 pm on Jan 8, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Does anyone reading this know whether the default for "Show friendly HTTP Error messages" (Internet Options-Advanced) has changed in MSIE7? It is un-checked on my system, but I do not recall changing it. It is most definitely checked by default on MSIE6 and earlier.
3:49 pm on Jan 8, 2007 (gmt 0)

5+ Year Member



Thanks for sharing the New Year's resolution. It's important enough to look at at least once a year, particularly if you are dynamic on your site. Things can always change, seemingly of their own accord at times. I didn't see any reference to the header checker on this site, so for anyone who hasn't checked yet, here you go. Happy New Year!

Server Header Check [webmasterworld.com]

3:53 pm on Jan 25, 2007 (gmt 0)

5+ Year Member



There are two seperate issues here:

Whether you include the "www" or not in your URL is not important as far as HTTP header status codes are concerned. The "www" part of your domain name only indicates to the DNS system which server should get the request. Most DNS systems are configured these days with your web server as the default and so your web server will get the request even if you omit the "www" part.

As far as HTTP response header status codes are concerned, you should return a 404 status even if you are using a custom 404 Error page. If you are re-directing to your custom 404 page then generally your web server will return 200 OK because it has actually found and returned a page that your script ultimately asked for.

I don't know which scripting language you are using for your CMS but in PHP you would include at the top of your 404 error page script something like:

<?php
header('Status: 404 Not Found', null, 404);
?>

Hope that helps.

12:06 pm on Jan 26, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Herein lies the problem...

Our cms provider doesn't consider it a problem, but we can pay them to change it if we want. However...

Translation: (almost) no-one using MSIE will see your own 404 pages. They will instead see Microsoft's idea of a friendly 404 page.

I may have a hard time explaining to my bosses/colleagues why paying our CMS provider to make the friendly 404s 'disappear' is a good thing!

FYI our cms is not php, but .NET.

12:30 pm on Jan 26, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Translation: (almost) no-one using MSIE will see your own 404 pages. They will instead see Microsoft's idea of a friendly 404 page.

That's not my experience (although I have not checked IE7).

All you need to do is make sure the 404 has a text length above (I think) 512 characters and it will show. 'Friendly' 404 pages are the result of Microsoft deciding that a 404 page with less than 512 characters is not giving enough information to the user. You can even use html comments to get over the 'Feature'.

2:02 pm on Jan 26, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Might it be worth offering people a quick and dirt way of checking the response code sent by their server?

The following Perl script will request a specified URI, and display the status code returned by the http server:

eg use:

perl script.pl http://www.example.com/page.htm

#!/usr/bin/perl

use LWP::UserAgent;
use HTTP::Response;

$url = $ARGV[0];
$ua = LWP::UserAgent->new;
$ua->agent("checking");

my $response = $ua->get($url);
printf "%s\n",$response->status_line;

Matt

2:44 pm on Jan 26, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Got anything equivalent in ASP Matt?
3:23 pm on Jan 26, 2007 (gmt 0)

5+ Year Member



You can use the Web Developer Extension Toolbar for Firefox to view the complete set of response headers including the Status Code.

It's available at [chrispederick.com ]

If you install it, the feature is under "Information >> View Response Headers"

3:58 pm on Jan 26, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks nwhiteley!

Duh - I had this all along, just hadn't spotted it!

8:39 pm on Jan 26, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



inbound:
All you need to do is make sure the 404 has a text length above (I think) 512 characters and it will show

Thanks, inbound: - new info for me.

The reference page appears to be:


Internet Information Services (IIS) 5.0 returns a status code of 500 for any response that is not handled by another 1xx, 2xx, 3xx, 4xx, or 5xx status code ... IIS 4.0...returns...errors with a status code of 200

Several frequently-seen status codes have "friendly" error messages that Internet Explorer 5.x displays ... these "friendly" error messages are only displayed if the response that is sent to the client is less than a specified threshold.

We now get:
friendly HTTP-status error messages are stored in the following registry key:
HKEY_LOCAL_MACHINE\
Software\
Microsoft\
Internet Explorer\
Main\
ErrorThresholds
(does not exist with MSIE7)

There is a name value pair (for example, "404", 128) for each of the errors ... second value is the byte size value ... Wininet.dll file determines if the HTML content attached to the HTTP error is a well designed Web page. This is based on the size of the page.
(never mind the quality, feel the width)

So, my previous advice to HelenDev needs to be modified...

HelenDev:

(custom 404 pages giving 200 OK status instead of 404) I asked our cms provider about this and they said that was the correct behaviour, and it shouldn't generate a 404.
That is correct advice for IIS 4, and dangerously wrong advice for anything else.

Our CMS provider seems to be suggesting that sending a 404 header would mean losing the friendly error pages...
That is correct advice if the custom-404 pages are smaller than 512 bytes, and nonsense otherwise.

MSIE7 behaviour appears not to be documented in this regard - has M$ abandoned this aspect of MSIE? (Yipee!)

 

Featured Threads

Hot Threads This Week

Hot Threads This Month