homepage Welcome to WebmasterWorld Guest from 54.204.68.109
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Visit PubCon.com
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
Check Server Headers
New Year's Resolution for all...
pageoneresults




msg:3206372
 12:30 am on Jan 3, 2007 (gmt 0)

Please, if you do one thing in the next 24 hours, check your server headers and be sure that the proper http status codes are being returned. As an added twist, find a tool that does recursive lookups. If all you are seeing is the destination response, you may be missing out on some important responses along the way to that final destination response.

I just caught a 302 loop on a particular domain. Using most server header tools, I can only see the final response which is a 302. But, it doesn't tell me that there is a loop occurring. :(

Please, double check and triple check your rewrite directives and make sure that proper server responses are being returned for ALL URI queries.

For those of you with a custom 404 in place, be sure to check the server header response on that one too. If you are returning a 200 status instead of a 404, there may be problems.

 

HelenDev




msg:3206691
 11:36 am on Jan 3, 2007 (gmt 0)

Thanks pageoneresults, just the topic I wanted to ask about :)

We use a cms which generates a custom 404 page if a page doesn't exist. This thankfully doesn't happen too often as our cms practically eliminates broken links, but this page does appear if a user types in a wrong url for example. I guess it would also appear if someone had bookmarked a page which subsequently disappeared, but at least 90% of our pages are probably permanent.

I was trundling around on ww the other day and came upon the header checking tool, and I tested our custom 404 page, and found that it was giving 200 OK status instead of 404. I asked our cms provider about this and they said that was the correct behaviour, and it shouldn't generate a 404.

Is this correct? Everything I've read here seems to suggest otherwise. How much of an issue is this and what problems could it cause? The site in question is a public sector one and we seem to do OK in Google, and bearing in mind that we don't generally have broken links or a fast turnover of pages, should we be concerned at all?

pageoneresults




msg:3206851
 3:04 pm on Jan 3, 2007 (gmt 0)

Is this correct?

If you test a URI that does not exist for your site, what is the status returned in the header? A 404 or a 200? If a 200 is returned for a URI that does not exist on your site, then it is not set up properly and may cause problems.

P.S. I'd also check the www vs. non-www and be sure one is 301'd to the other. I see some sites using a 302 when they should probably be using a 301. ;)

Frank_Rizzo




msg:3206944
 4:31 pm on Jan 3, 2007 (gmt 0)

re: www vs non

If in the browser URL address I enter http://www.example.co.uk/ that is fine.

If I enter http://example.co.uk/ I get served that page with status 200.

I guess that is not correct as surely it should redirect to www.widgets.co.uk?

In httpd.conf I have this:


RewriteCond %{HTTP_HOST} ^example\.co\.uk
RewriteRule ^/(.*)$ http://www.example.co.uk/$1 [R=permanent,L]

So it looks as if something is not working?

[edited by: encyclo at 10:44 pm (utc) on Jan. 4, 2007]
[edit reason] switched to example.co.uk [/edit]

jdMorgan




msg:3206979
 4:49 pm on Jan 3, 2007 (gmt 0)

HelenDev,

> I asked our cms provider about this and they said that was the correct behaviour, and it shouldn't generate a 404.

Well, they are wrong. As PR1 clearly states, any URL that does not resolve to an existing page should return a 404 status -- end of story, no wiggle-room, period, full stop. Otherwise, there is nothing to tell search engines not to index and list that URL in their search results. You risk you site's theme being diluted, and over time, shifting from its original subject to the theme, "This page not found".


Frank_Rizzo,

> surely it should redirect to www.example.co.uk

Yes it should. Is there a "RewriteEngine on" directive somewhere above that rule, as required to enable rewriting?

Jim

[edited by: encyclo at 10:44 pm (utc) on Jan. 4, 2007]

Frank_Rizzo




msg:3207088
 6:24 pm on Jan 3, 2007 (gmt 0)

Here's the full section dealing with that stuff:


<Directory "/home/widgets/public_html">
Options FollowSymLinks

<Files ~ "^.*$">
order allow,deny
allow from all
deny from 198.133.178.0/23 #central new mexico college
#many other's listed here - removed for display purposes
</Files>

RewriteEngine on
RewriteCond %{HTTP_HOST} ^example\.co\.uk
RewriteRule ^/(.*)$ http://www.example.co.uk/$1 [R=permanent,L]
RewriteCond %{SCRIPT_FILENAME} ([^/]+)\.wmv$
RewriteRule ^.*$ http://www.example.co.uk/misc/misc.html [R=permanent,L]
</Directory>

The script filename / .wmv section is a workaround for a self loading video file. Is this causing the redirect not to work?

[edited by: encyclo at 10:44 pm (utc) on Jan. 4, 2007]

alfaguru




msg:3208571
 9:35 pm on Jan 4, 2007 (gmt 0)

Frank, the slash at the beginning of your rewrite rule is not wanted. It should be like this:


RewriteRule ^(.*)$ http://www.example.co.uk/$1 [L,R=301]

[edited by: encyclo at 10:45 pm (utc) on Jan. 4, 2007]

HelenDev




msg:3212018
 12:33 pm on Jan 8, 2007 (gmt 0)

Thanks for the clarification, that's what I thought :)

Presumably you can still have custom error pages and send a 404 header? Our CMS provider seems to be suggesting that sending a 404 header would mean losing the friendly error pages...

AlexK




msg:3212032
 1:04 pm on Jan 8, 2007 (gmt 0)

HelenDev:
Our CMS provider seems to be suggesting that sending a 404 header would mean losing the friendly error pages...

Not fully accurate.

MSIE has "friendly" (sic) 404 pages.

Translation: (almost) no-one using MSIE will see your own 404 pages. They will instead see Microsoft's idea of a friendly 404 page.

AlexK




msg:3212038
 1:09 pm on Jan 8, 2007 (gmt 0)

Does anyone reading this know whether the default for "Show friendly HTTP Error messages" (Internet Options-Advanced) has changed in MSIE7? It is un-checked on my system, but I do not recall changing it. It is most definitely checked by default on MSIE6 and earlier.

smells so good




msg:3212163
 3:49 pm on Jan 8, 2007 (gmt 0)

Thanks for sharing the New Year's resolution. It's important enough to look at at least once a year, particularly if you are dynamic on your site. Things can always change, seemingly of their own accord at times. I didn't see any reference to the header checker on this site, so for anyone who hasn't checked yet, here you go. Happy New Year!

Server Header Check [webmasterworld.com]

nwhiteley




msg:3231575
 3:53 pm on Jan 25, 2007 (gmt 0)

There are two seperate issues here:

Whether you include the "www" or not in your URL is not important as far as HTTP header status codes are concerned. The "www" part of your domain name only indicates to the DNS system which server should get the request. Most DNS systems are configured these days with your web server as the default and so your web server will get the request even if you omit the "www" part.

As far as HTTP response header status codes are concerned, you should return a 404 status even if you are using a custom 404 Error page. If you are re-directing to your custom 404 page then generally your web server will return 200 OK because it has actually found and returned a page that your script ultimately asked for.

I don't know which scripting language you are using for your CMS but in PHP you would include at the top of your 404 error page script something like:

<?php
header('Status: 404 Not Found', null, 404);
?>

Hope that helps.

HelenDev




msg:3232705
 12:06 pm on Jan 26, 2007 (gmt 0)

Herein lies the problem...

Our cms provider doesn't consider it a problem, but we can pay them to change it if we want. However...

Translation: (almost) no-one using MSIE will see your own 404 pages. They will instead see Microsoft's idea of a friendly 404 page.

I may have a hard time explaining to my bosses/colleagues why paying our CMS provider to make the friendly 404s 'disappear' is a good thing!

FYI our cms is not php, but .NET.

inbound




msg:3232723
 12:30 pm on Jan 26, 2007 (gmt 0)

Translation: (almost) no-one using MSIE will see your own 404 pages. They will instead see Microsoft's idea of a friendly 404 page.

That's not my experience (although I have not checked IE7).

All you need to do is make sure the 404 has a text length above (I think) 512 characters and it will show. 'Friendly' 404 pages are the result of Microsoft deciding that a 404 page with less than 512 characters is not giving enough information to the user. You can even use html comments to get over the 'Feature'.

Matt Probert




msg:3232790
 2:02 pm on Jan 26, 2007 (gmt 0)

Might it be worth offering people a quick and dirt way of checking the response code sent by their server?

The following Perl script will request a specified URI, and display the status code returned by the http server:

eg use:

perl script.pl http://www.example.com/page.htm

#!/usr/bin/perl

use LWP::UserAgent;
use HTTP::Response;

$url = $ARGV[0];
$ua = LWP::UserAgent->new;
$ua->agent("checking");

my $response = $ua->get($url);
printf "%s\n",$response->status_line;

Matt

HelenDev




msg:3232830
 2:44 pm on Jan 26, 2007 (gmt 0)

Got anything equivalent in ASP Matt?

nwhiteley




msg:3232863
 3:23 pm on Jan 26, 2007 (gmt 0)

You can use the Web Developer Extension Toolbar for Firefox to view the complete set of response headers including the Status Code.

It's available at [chrispederick.com ]

If you install it, the feature is under "Information >> View Response Headers"

HelenDev




msg:3232919
 3:58 pm on Jan 26, 2007 (gmt 0)

Thanks nwhiteley!

Duh - I had this all along, just hadn't spotted it!

AlexK




msg:3233308
 8:39 pm on Jan 26, 2007 (gmt 0)

inbound:
All you need to do is make sure the 404 has a text length above (I think) 512 characters and it will show

Thanks, inbound: - new info for me.

The reference page appears to be:


Internet Information Services (IIS) 5.0 returns a status code of 500 for any response that is not handled by another 1xx, 2xx, 3xx, 4xx, or 5xx status code ... IIS 4.0...returns...errors with a status code of 200

Several frequently-seen status codes have "friendly" error messages that Internet Explorer 5.x displays ... these "friendly" error messages are only displayed if the response that is sent to the client is less than a specified threshold.

We now get:

friendly HTTP-status error messages are stored in the following registry key:
HKEY_LOCAL_MACHINE\
Software\
Microsoft\
Internet Explorer\
Main\
ErrorThresholds
(does not exist with MSIE7)

There is a name value pair (for example, "404", 128) for each of the errors ... second value is the byte size value ... Wininet.dll file determines if the HTML content attached to the HTTP error is a well designed Web page. This is based on the size of the page.
(never mind the quality, feel the width)

So, my previous advice to HelenDev needs to be modified...

HelenDev:
(custom 404 pages giving 200 OK status instead of 404) I asked our cms provider about this and they said that was the correct behaviour, and it shouldn't generate a 404.
That is correct advice for IIS 4, and dangerously wrong advice for anything else.

Our CMS provider seems to be suggesting that sending a 404 header would mean losing the friendly error pages...
That is correct advice if the custom-404 pages are smaller than 512 bytes, and nonsense otherwise.

MSIE7 behaviour appears not to be documented in this regard - has M$ abandoned this aspect of MSIE? (Yipee!)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved