Tracking of 400 and 500 Errors

Are you being Proactive or Reactive?

[pageoneresults]

For many years, I've found myself dreading the task of scanning raw log files for errors, etc. Or even worse, waiting for a user to report an error. :(

These days, things have changed. I'm responsible for a fairly large distance education environment and one of the things we are now being very proactive in our approach to are errors! When you have 500+ students accessing streams, quizzes (1,000's of questions and answers), there are all sorts of things that creep up.

We've developed a system on our Windows server(s) that now reports "all" errors to the system administrators. This is done via an ASPEmail script and works like a charm. The emails are formatted a specific way to allow recipients to easily filter and address them.

We've set up custom 400 and 500 pages for the visitors that give them friendly error messages along with details of the error if they wish to keep a copy of it. The message tells them that the system administrators have been informed of the error and it will be addressed within 24 hours, that's our normal turnaround time.

If there is one thing I've learned from this entire process, that is the importance of being Proactive vs. Reactive when it comes to dealing with errors. Those 500 errors will kill the visitor experience and getting notified immediately when they occur is something that every web developer should have easy access to.

Is this something you are concerned with? Do you have systems and methods in place to effectively manage site errors? How do you do it?

[kolin]

this is actually something i have done with one of my sites.

if an error occurs, i serve up a specific 500/40x page and get the web application to instantly send me an error email, so i can quickly fix the problem.

it's currently a horribly reactive method to sit and wait for problems to happen, but the most thorough testing will not get all issues within the stupidly small timescales us web developers are given. and it's better than somebody emailing in a 500 server error to somebody who then starts to panic. i'd rather it go straight to me who wont panic, and just get it fixed.

[giggle]

Now that's what I call timing!

Yesterday I set up a 500;100 to catch ASP errors and give the user a soft bump should an error occur. I set up a new page that looked nice for the user but also sent an email and SMS'd me with details of the problem. Let's just say that I was busy last night sorting out some coding issues I would not have been aware of had this new page not been there.

I kind of feel like I'm being both proactive (handing an error situation in a positive, controlled way) and reactive (in the sense that now I can now react quickly to errors and correct them). Well, that's my interpretation anyway.

Does anyone know how I can catch the http status code (and any additional details) in ASP so that I can create one page to handle all the various situations.

Thanks

Mick

[RonPK]

One of my clients has an everchanging extranet that runs on PHP and MySQL. Error handling & reporting is a key feature of the system. All PHP and MySQL errors are reported by email to me, and the user gets to see a custom error page. I find that having a good error handling module also benefits development as it makes it easier to find the cause of errors.

[g1smd]

It would be good to see a quick list of the sort of problems that had to be fixed, so that others don't make the same mistakes...

[JS_Harris]

I've done this with my own servers, great post and a good idea for all webmasters to think about.

I have some sites hosted with others however and I wish they would implement this too. They don't include the errors in their reporting either but my own tracking shows them. I'm all for this change.

[souravamant]

Yes indeed, I run a very small blog and I have been using this for a while with great results..

Sourav

[pageoneresults]

It would be good to see a quick list of the sort of problems that had to be fixed, so that others don't make the same mistakes.

The list could be quite long and specific to the application. I've found the process of tracking 400 and 500 errors to be a very "eye-opening" experience and one that I would definitely recommend to all involved with development.

Just think...

Edward, I ran into an error yesterday while doing this...

No problem, we've already addressed it and you should be good to go.

Proactive is the way to go. They say for every one (01) error reported, there are nine (09) others that are not being reported. That should be enough incentive in itself to look into something like this. ;)

[milanmk]

I would like to point one important part called error logging. I do have all the notification methods in place but I make sure to dump most of the client and server information relating to that error to an error log file, which is kept outside of root folder and secured by Apache directives.

Many times it happened that the notification emails been deleted once the error has been rectified and then suddenly I found some related bug but do not have any supportive data to relate it to the previous error.

Therefore, I think error logging is also crucial part that will allow you to inspect past errors and help you in rectifying future ones.

Milan

[klown]

Great topic, for some reason I never thought to record the errors I was receiving. Anyhow I wrote a bit of PHP to email me info on my errors. The code should work on any site, you just need to change the email address and make a bit of a modification to your .htaccess file (adding on?errorcode=404 etc.). Hope this is useful for somebody.

The script will email you:
IP
Error code and definition
Browser version
Date and Time
Referrer Page
Website the error occurred on

<?php
$referid=$_SERVER['HTTP_REFERER'];
$url="http://".$_SERVER['HTTP_HOST'];
if (getenv("HTTP_CLIENT_IP"))
$ip = getenv("HTTP_CLIENT_IP");
else if(getenv("HTTP_X_FORWARDED_FOR"))
$ip = getenv("HTTP_X_FORWARDED_FOR");
else if(getenv("REMOTE_ADDR"))
$ip = getenv("REMOTE_ADDR");
else
$ip = "UNKNOWN";
$datetime=date("r");
$browser=$_SERVER['HTTP_USER_AGENT'] ;
$errorcode=$_GET['errorcode'];
switch ($errorcode)
{
case 400:
$definition="Bad Request";
break;
case 500:
$definition="Internal Server Error";
break;
case 403 :
$definition="Forbidden";
break;
case 404:
$definition="Not Found";
break;
case 408:
$definition="Request Timeout";
break;
}
$subject="$url Error $errorcode";
$body="URL: $url
Error: $errorcode $definition
Referer: $referid
IP: $ip
Browser: $browser
Datetime: $datetime
";
$email="[e-mail]";
mail ($email,$subject,$body);
?>

[jdMorgan]

I'd like to add a note of caution here, although I agree that the proactive approach has a lot of merit.

When designing your notification and/or logging system, keep it simple. You want to minimize the dependencies of the error-handler. For example, if you have a problem with your PHP configuration which causes a serious problem running PHP code under certain circumstances, and you then try to use PHP to report or log that error, then you'll get knock-on errors, with a server error causing another server error, etc.

So error handling --especially that for the 500-Server error-- should be kept as simple as possible.

Jim

[travelorama]

Lots of great ideas here.

How about logging errors off-site by serving visitors a tiny JS? Even if PHP went down, a plain HTML page could do the trick.

Just my 2c

[jgar]

OK, while were on the subject, how about tracking image-loading problems ... the page loads, but an image is not available - I would say this may be a fairly common error that for the user makes quite a difference, but maybe is a difficult error to track - we use ASP, any ideas?

[giggle]

JGar

I recently set up a 404 that emails me when missing pages are encountered. A side effect of that is that I also get emails when pictures are missing. It doesn't redirect the page to the 404 page, the page works normally and I still receive the missing pictures email.

Mick

[denisl]

I have got to say thanks to KLOWN above for that code to send email reports from my custom 404 pages - I have found this unbelievably useful.

I added this to get the full the path of the page that was not found

$path=$_SERVER['REQUEST_URI'];

[Bewenched]

Giggle -
in response to missing images. i have a nifty little script that makes sure the image is there, if not show a transparent gif. it's in VB script/asp if any one is interested pm me. The only problem with it is that it is case sensitive to the file name so if it's calling whatever.jpg and the file is really named WhatEver.jpg it will show the alternate image. I haven't found a work around for that yet.

[jixi]

You could use crawlscore to identify the problems in the first place but's a good idea to log them going forward.

hth.