htaccess and Error 403 vs error 500

[TorontoBoy]

I have different web sites all in directories and use a single htaccess in public_html, on a shared hosting server. My htaccess has some RewriteCond (usually UA and referrer), a lot of SetEnvIf, and a lot of Deny x.x.x.x/x.

When does the server return a 403 and when does it return a 500. By knowing this it would be somewhat easier for me to troubleshoot, for example, when I see a ban in my log that should not be banned.

Sometimes I find it difficult to figure out why something was banned, for example a 500. Checking the IP, or the IP range is easy enough, but sometimes I find an entry that I simply cannot track down.

Does Apache cache the htaccess in any way for efficiency? It seems like some of my bans were from months ago, which I have since unbanned, but I don't see the change reflected in the logs.

Thanks in advance.

[lucy24]

The server returns a 500 when you have made a mistake. 500-class errors are serverspeak for “It isn’t you, it’s me.” ALWAYS do a test fetch each time you upload a changed htaccess, no matter how trivial the change. (Even then, some changes--like a misplaced comma in access-control rules--will only affect log format, so you may not notice the mistake at once.) If only some requests receive a 500, it is most likely because the mistake is in a rule or RewriteCond that isn't evaluated on every request.

.htaccess is not cached. That is one of the key differences between putting something in htaccess and putting it in config; an htaccess file is read on every request.

[phranque]

the 5xx server response isn't a "ban", it's a Server Error".
see https://tools.ietf.org/html/rfc7231#section-6.6

whenever you see a 500 response you should check your server error log file for clues.

a 4xx is technically a "Client Error".
see https://tools.ietf.org/html/rfc7231#section-6.5

[TorontoBoy]

Here is an example where I was expecting a 403 , but received a 500.

144.217.15.xxx [06/May/2018:19:46:11 GET /robots.txt HTTP/1.1 500 - - Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
144.217.15.xxx [06/May/2018:19:46:20 GET / HTTP/1.1500 - - Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)

I was expecting a 403 due to a UA rule (below) but received a 500 instead? I do not ban the IP. Can someone shed any light?

SetEnvIf User-Agent SiteExplorer keep_out

158.69.252.xxx [06/May/2018:09:03:03 GET /robots.txt HTTP/1.1 500 - - Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
158.69.252.xxx [06/May/2018:09:03:14 GET / HTTP/1.1 500 - - Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
167.114.219.xx [09/May/2018:04:54:43 GET /robots.txt HTTP/1.1 500 - - Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)
167.114.219.xx [09/May/2018:04:54:56 GET / HTTP/1.1 500 - - Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)

In addition to the above UA ban, this also has an IP ban on 158.69.0.0/16 and 167.114.0.0/16 I was expecting a 403 but received a 500. Why? Thanks to all

[keyplyr]

It's possible someone has assigned 500 error to handle 403 error.

[lucy24]

I was expecting a 403 due to a UA rule (below) but received a 500 instead?

Do all intended 403s receive a 500 instead, or only some of them? If it’s all of them, the next question becomes: Is your host doing it on purpose (why, for ### sake?) or does it reveal some underlying ineptitude on their part? If it’s only some of them, we’ll need to take a closer look and try to figure out the variable.

There are plenty of available 400-class responses without resorting to a spurious 500. My host, for example, returns a 418 (Teapot Error) on requests blocked by mod_security, so you can tell which lockouts are theirs and which are yours. (418 takes priority when both could apply.)

[TorontoBoy]

That would make a lot of sense. Hence my confusion as to the origins of some of my 500s. The vast majority do come out as 403s, but not all. Thanks.

I am trying to logically figure this out.

[phranque]

I was expecting a 403 , but received a 500.

whenever you see a 500 response you should check your server error log file for clues.

[Iskandrian]

My host has begun doing this - that is, reporting Rewrite blocks as 500s instead of 403s, which is why I switched to SetEnvIf, which immediately corrected it back from 500 to 403. Those blocking sections still using Rewrite still report 500s instead of 403s.

[TorontoBoy]

My server error log, from cPanel, does not show error codes, just IP address and error messages such as

[Thu May 10 18:07:07 2018] [error] [client 148.251.176.xx] client denied by server configuration: /home/example.com/public_html/subdir/2009

I only see the error codes in my raw access log.

I use both SetEnvIf and some RewriteCond. In my case above the UA rule is only SetEnvIf. SetEnvIf usually gives me 403s, but in this case returns 500s.

SetEnvIf User-Agent SiteExplorer keep_out

[phranque]

are there any .htaccess files in the relevant subdirectories of public_html?

[TorontoBoy]

yes, there are .htaccess files in every subdirectory, but they have no SetEnvIfs, RewriteConds or deny from IP bans. They are bog standard from WP, Drupal and other CMS.

[keyplyr]

I've seen shared hosting accounts set up like that.

[phranque]

SetEnvIf usually gives me 403s, but in this case returns 500s.

are your current logs still showing 403s for all the other SetEnvIf-based blocks?

[lucy24]

“Client denied by server configuration” is the standard Apache text for any and all 403 responses. It never says anything more, regardless of your logging level. (I’ve experimented in MAMP.) In particular, it won’t say which mod issued the 403. Are you saying that you see this response in cPanel logs even when the access logs say 500? Yuk. That’s certainly not very helpful.

Error logs never show response codes; they show what the problem was, not what the server did about it. Another one you'll see periodically is “File does not exist” which may come through in access logs as a 404 error, if the server is looking for a requested file--or it may not, for example if the server is looking for a custom error document that doesn't exist. Genuine 500-class errors generally give the most useful information, since the server is telling you about its own inner workings. If you have mod_security, that also gives lovely detailed error reports, showing what string is being matched against what, rather like mod_rewrite logs (which we don't have access to in shared hosting). Unlike most errors, it even says how it handled the issue.

Remember that mod_setenvif itself does not issue 403s. Those come from mod_authzthingummy (exact name depends on your Apache version) when you say
Deny from env=whatever
As far as the server is concerned, it is no different from
Deny from 52

[TorontoBoy]

yes, almost all of my SetEnvIfs all show 403s, with the exception of the odd 500. I have been struggling for a long time to figure our when I should get a 403 and when I should get a 500. It seems I cannot figure out the logic.

I have multiples sites on a shared host provider, all sites in subdirectories. I have a single large htaccess in public_html, where all my SetEnvIfs go. The vast majority of 500s I can trace back to a UA rule or an IP ban. I believe these should be 403s instead, but they return 500s. I continue to dig and analyze why.

@lucy I have way more 403s and 500s than show in my error log. The log shows only the last 300 errors. I estimate I only see 5% of my 403/500s in my error log. I therefore need to rely on my raw access log.

Remember that mod_setenvif itself does not issue 403s.

What does this then return as an error number?

[lucy24]

What does this then return as an error number?

Nothing. The point is that mod_setenvif in and of itself does not do anything about access. Its only function is to set environmental variables, which can later be used by other mods, including mod_authdoodad and mod_rewrite, both of which can make access decisions.

You can't download your entire error logs, the way you can download access logs? What rotten luck. Not that, er, I ever do look at error logs, except when I'm investigating some particular item--but I can if I want to. They're saved on the server for the same time period as access logs.

[keyplyr]

They're [error logs] saved on the server for the same time period as access logs.

We'll see if that's still true after May 25.

[TorontoBoy]

I have spent a couple more hours on my 403 vs 500 issue. There are 3 variables I am observing: deny IP ban, SetEnvIf, and RewriteCond. I have no SetEnvIf and RewriteCond: they are mutually exclusive. Recently I have noticed for every 100 403s I have only 4.8 500s.

deny IP ban, no SetEnvIf or RewriteCond: 403
deny IP ban and RewriteCond ban : no SetEnvIf: 500
deny IP ban and SetEnvIf ban: ?, not seen yet
RewriteCond ban, no deny IP ban or SetEnvIf: 500
SetEnvIf ban, no deny IP ban or RewriteCond: 403

4:403, 5:500 I hope this "chart" is not too difficult to read.
______________4 5 ? 5 4 - - -
deny IP ban__ Y Y Y N N Y N N
ReWriteCond N Y N Y N Y N Y
SetEnvIf____ N N Y N Y Y N Y

I am going to try to replace some of the well used ReWriteCond and redo them in SetEnvIf, and see what happens.

@lucy Thanks for the clarification. The SetEnvIf sets the environment variable and nothing more, but is followed by the deny, which does the ban, right?

order allow,deny
allow from all
deny from env=keep_out

so technically the deny is the statement that does the ban.

[lucy24]

technically the deny is the statement that does the ban.

Exactly. That means you should notice absolutely no difference between IP-based Deny and env-based Deny. If you do find a difference, you may have a weird server.

[TorontoBoy]

After changing my referrer ReWriteCond rules to SetEnvIf, my error 500s are significantly reduced. I can say, though do not know why, ReWrite Cond produces 500s on my shared host. ReWriteCond rules do their intended block, so the actual result of the change, in real life terms is minimal.

[TorontoBoy]

I confirm that my ReWriteCond statements were the cause of my 500s. After conversion to SetnvIf my server now returns 403s, as they should. I now only have a fraction of previous days. My shared host provider could not help me answer this question for over a year. While the block is the same, why would the server do this?

There is also no correlation between my cPanel errors page and the number of 500s I see in my raw access log.