Fixing 500 server errors

What is causing them?

[grandma genie]

OK, let's try this again.

I have been finding 500 errors in the logs. Like this:

183.88.80.nnn "GET / HTTP/1.1" 500 - "www.example.com/example" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.75 Safari/537.1"

This referer is blocked in htaccess like this:

# Return 403-Forbidden response for bad referers
RewriteCond %{HTTP_REFERER} www\.example\.com [NC]
RewriteRule .* - [F]

I want a 403, not a 500. This is the only referer block I have. My htaccess file is set up like this:

Options All -Indexes
Options +SymLinksIfOwnerMatch
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /example/
more RewriteRules here for changing my numerical URLs to words.
</IfModule>
#
#301 redirects
#Listing of my 301 redirects here like this
RewriteRule ^example.html$ website URL here/example/example.html [R=301,NC,L]
#
# Return 403-Forbidden response for bad referers
RewriteCond %{HTTP_REFERER} www\.example\.com [NC]
RewriteRule .* - [F]
#
# Return 403-Forbidden response for hotlinked image requests
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?example\.com [NC]
RewriteRule \.(jpe?g|gif|bmp|png|ico)$ - [F]

Can anyone see if there is something here that might cause the 500 error? Is the htaccess set up incorrectly?

Thank you for any help.

-- GG

[edited by: incrediBILL at 7:03 am (utc) on Aug 11, 2012]
[edit reason] fixed URLS, use Example.com [/edit]

[lucy24]

You need to look at your error logs. Log level is out of your control, so you don't know how much or how little they will say. But they'll always give some information about 500-class errors.

Do you have a custom error page that's set with the ErrorDocument directive, or do you use the host's default filenames like "missing.html" or "forbidden.html"? I had a very belated "D'oh!" moment a few months ago as I realized that the config file must have a whole set of <Files> directives involving these specific names-- so if you use a different name you have to add conditions.

In particular, in rules that result in [F], like this

RewriteCond %{HTTP_REFERER} www\.example\.com [NC]
RewriteRule .* - [F]

you may need to put in a second Condition like

RewriteCond %{REQUEST_FILENAME} !my403\.html

to keep from going around in circles. I've copped out by putting a single Rule at the very beginning:

RewriteRule (boilerplate|pictures/(hotlink|smallgifs/onedot)) - [L]

That intercepts anything that has already been rewritten. It also gives a free pass to anyone asking for these files by name, so there are drawbacks. But in my case I don't think they could actually do anything with this group of files.

[wilderness]

I suspect the 500 is related to the missing ErrorDocument.
It's quite common.


You may also try the following closing line (modified for your path and/or file names):

RewriteRule (custom403\.html|robots\.txt)$ - [F]

FWIW, in refers your not required to use the complete domain name and/or the referring sub-text, although that is perfectly acceptable.

RewriteCond %{HTTP_REFERER} www\.example\.com [NC]

You may use any portion of the domain name and/or referring sub-text

#if refer contains example
RewriteCond %{HTTP_REFERER} example [NC]

[g1smd]

You need to exclude (internal) requests for the ErrorDocument from being blocked to avoid the display of the error page from being blocked.

[grandma genie]

The host had two files in the public folder when I moved to that host that were called 404.shtml and 403.shtml. But they were causing issues with the index.php file I had, so I removed them. The coding in them was just a header redirect.

Lucy, the error log showed that the server was looking for those error pages. So, I assume I need to replace them. And I assume I need to include the ErrorDocument info in the htaccess file.

Can you recommend a site that offers custom error page coding?

And, Don, if I did this:

RewriteRule (403\.shtml|404\.shtml)$ - [F]

That would keep the server from looking for those custom pages and just serve up the standard error messages? That way I wouldn't need the custom pages. Or do you recommend having those custom pages?

-- GG

[g1smd]

The [F] flag would cause it to look for the file defined by the ErrorDocument directive.

If that file were "403.shtml" then you'd get Error 500 instead.

You mention the error files previously contained a "header redirect". That is always a bad idea.

[wilderness]

The host had two files in the public folder when I moved to that host that were called 404.shtml and 403.shtml.

Or do you recommend having those custom pages?

gg,
Most shared hosting servers add these files by default.
Many shared hosts even server their own 403s and 404s in higher-than-root cpaacity, unless you disable theirs and define your own via CP.

Hosts 403s/404s basically amount to advertising for the host. The same host that your paying for you hosting.
I had a host for many years (8 or 9) that did not utilize this practice, and I had no ErrorDocuments in place, thus a standard 403/404 was served by either the users browser or some standard Apache doc.

This host-practice required me to create and initiate my own ErrorDocuments.

[grandma genie]

I just had a chat with the host. What a waste of time that was. Would it be OK if I posted the code I found for a custom 403 page and see what you think? Or is that a no no?
-- gg

[lucy24]

The host had two files in the public folder when I moved to that host that were called 404.shtml and 403.shtml. But they were causing issues with the index.php file I had

Hmmmm. I think we need to know a little more. There shouldn't be any connection between an index file and an error document.

Among other things, that "shtml" extension implies the files are including stuff from elsewhere. Advertising, possibly. But if the host didn't want you to edit the files, they wouldn't have been accessible to you in the first place.

the error log showed that the server was looking for those error pages. So, I assume I need to replace them

You generally have two options in shared hosting.

#1 Use files with the default name in the default location-- but format them nicely to match the rest of the site. (wilderness, I think gg's and my hosts may be one nano-notch above yours in this respect. The filename and location are built in-- but it's in your own userspace, so you can change the content of the document.)

#2 Use error documents with your own choice of name and location, accompanied by the "ErrorDocument" line in htaccess.

When there is a 500-class error, the server either can't or doesn't look at your htaccess, so it only looks for the default file specified in the config file. I've seen the same type of error myself-- but it's not the normal response to a 403. It only happens when
:: ahem ::
:: cough-cough ::
I've made a mistake somewhere.

[phranque]

or you have the option of reverting to the default hard-coded apache error message sent with the error:

ErrorDocument 403 default

[wilderness]

or you have the option of reverting to the default hard-coded apache error message sent with the error:

many thanks phranque.

making this change in my htaccess has no effect, due to the higher-path implementation via CP to remove their settings.

Unfortunately my "el cheapo host" provides the following vague reference:

Initially, all error pages are set to "Default". To set a custom error page, choose "Custom URL" and enter the path to the corresponding error page in the field. If you haven''t created one yet, do this first and then get back to the Subdomain Manager section to update the settings.

What they do NOT explain is that their "default" contains their own custom advertising and a non-default Apache 403.

[lucy24]

Oh! That's even horribler than I thought. If you use the ErrorDocument {blahblah} directive in htaccess is it simply not recognized? or do you get a 500-class error? or, heh-heh, does it work perfectly well but they don't want you to know it exists?

Every time I see this thread title I think "Oh, man, that's a lot of errors" -- especially when there's a nearby thread called "301 redirects" ;)

[g1smd]

::snigger:: I had the same thoughts.

[phranque]

only your host can fix this for you.
in order to override ErrorDocument directives in .htaccess you need the AllowOverride FileInfo directive in the apache configuration:
http://httpd.apache.org/docs/current/mod/core.html#allowoverride

without cooperation from the host your only option is finding and overriding all the host's default error pages.

[grandma genie]

Oh, I had a feeling this was where we were going with this. My host is utterly worthless. I doubt they would know what a config file was. I suspected the solution existed in a directory I could not access. Oh woe is me!

I actually removed all my referer blocks and only left the one, so I would know if I had made an error somewhere, but no, even that one block gets the 500 error. I guess a 500 error is better than getting a 200. The only blocks that work are the IPs. Robots.txt works, too. But user agent blocks and referer blocks get the 500. If I put those blocks in the root htaccess, the server doesn't even see them. I have to put them in the public folder htaccess file.

By the way, I did finally get those 302 redirects to become 301s. I just had to add a line of code in the index.php file. At least that is working.

This is my fifth or sixth host and they are all the same. I can understand why they won't bend over backward for some tiny $9.95 a month customer.

I appreciate all your help. You are all treasures. The only place I have ever been able to find help is here at webmasterworld. Thank you very much.

[wilderness]

gg,
FWIW, you shouldn't interpret all this rambling as the answer to your inquiry. Most of this thread has no correlation to your issues.

Have you tried this method?
SetEnvIf Referer ^http://www.example.com keep_out

My htacess is very, very large, while utilizing refers, UA blocks, and a multiple of other methods, which all function fine and as intended.

My ErrorDocuments also function, however their configuration must be defined in CP.
I did not copy and paste the complete instructions, which also sstates to include in htaccess.

BTW, I pay less per month than you, in fact, I pay one-fourth of what I paid for 8-9 years and for that same price get multiple domains hosted.
The inconvenience of the ErrorDoc's is trivial. My host provides excellent support and has worked with me on some key issues.

[phranque]

have you tried rolling your own 403.shtml file?

[wilderness]

phranque,
I sent gg the html for a 403 that I use, however did not hear back.

And, Don, if I did this:

RewriteRule (403\.shtml|404\.shtml)$ - [F]

That would keep the server from looking for those custom pages and just serve up the standard error messages? That way I wouldn't need the custom pages. Or do you recommend having those custom pages?

gg, this line was meant as example.
Simply change the file names to your custom pages.

[lucy24]

RewriteRule (403\.shtml|404\.shtml)$ - [F]

That would keep the server from looking for those custom pages and just serve up the standard error messages?

As g1 said first time around: No. It would only prevent the server from getting and displaying the error message, resulting in a fresh 403 error, resulting in another attempt to serve the 403.shtml document, resulting in a 403 error {next eight iterations snipped} ultimately leading to a 500 error.

Been there. Done that.

So if you have any rules that deny access to something with the extension used by your ErrorDocuments, make sure you have a supplementary rule that gives an exemption for those documents. It can be either an initial [L] to short-circuit further blocks, or something in a <Files> directive.

[wilderness]

gg, this line was meant as example.
Simply change the file names to your custom pages.

[grandma genie]

I just picked up the messages from Don. I will try those suggestions. I'll let you know what happens.

By the way, I have another little site hosted by the same host, so was able to pick up their standard 404.shtml file (there was none for 403s) and took a look at it. I think they coded it wrong. It is html code; has the beginning html, head, title, body codes, but does not have the ending body and html codes. I wouldn't think it would work without them. So, I added them and put that file in my other site, added the needed coding to the htaccess file, cleared the cache and tried asking for a non-existant page. But the 404 "file not found" page I got was not the file that is in the 404.shtml page. It appears to be the standard Apache 404 error message. Don't know why the server did not see the custom page. I put it in both the root folder and the public folder. It didn't make any difference.

Yesterday I put all my user agent blocks and referer blocks in the public htaccess file. They are all getting 500s. I'm gonna try Don's ideas.

I'm going to experiment a bit and see what happens. I'll let you know.

Thanks again.

[lucy24]

has the beginning html, head, title, body codes, but does not have the ending body and html codes. I wouldn't think it would work without them.

Do you mean that the words
</body>
</html>
are missing? Strange but true: Not only will their absence not prevent the page from running normally-- they won't even prevent the page from validating.

Er. If that was too many negatives: The page will display normally. It will pass validation.

80,000 guesses how I know this.

[grandma genie]

Still working on those files, but this method is working:

SetEnvIf Referer ^http://www.example.com keep_out

I have tested it on both user agents and referers and the server is now giving out 403s. Server doesn't seem to like the Mod Rewrites but has a thing for SetEnvIf. Thank you, Don.

And Lucy, isn't it amazing that my host would actually provide a custom 404 error page that was coded incorrectly. Doesn't that give you a warm, fuzzy feeling?

Things are looking up.

[g1smd]

Don't know why the server did not see the custom page.

It doesn't matter where you put it, you always need to refer to it using the ErrorDocument directive in the htaccess file. This reference must include path and file only, NOT protocol or hostname.

[wilderness]

It doesn't matter where you put it, you always need to refer to it using the ErrorDocument directive in the htaccess file. This reference must include path and file only, NOT protocol or hostname.

g1smd,
I sent her a communication suggesting adding of the ErrrorDocument lines, however it may have fallen through the cracks.

[Leosghost]

Do you mean that the words
</body>
</html>

Words?..Tags shirley? Tags ..

[grandma genie]

Files now working correctly, thanks to wilderness. Many thanks!

Don't think I need the custom pages now, but have it set up to accept them if I change my mind. Yes, there is an ErrorDocument directive in htaccess with the correct path. Commented out for the time being.

Attacking this project worked best by reducing the files down to minimum and then rebuilding them, with the help of the experts here at webmasterworld. Thanks again.

For someone who is moving from one host to another, remember, the server setup may be different, and an htaccess file might work fine with one host, but not with another.