RewriteCond vs. BrowserMatchNoCase

options to block bad bots

[smallcompany]

Hi,

Errors in Apache logs:

AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.

After tracing it back, it looked like "bad bots" being redirected to a custom 403 page were causing this. Server support confirmed it, and I found references about similar cases. That time related config in .htaccess was:

ErrorDocument 403 /error403.php
RewriteCond %{HTTP_USER_AGENT} BadBot1 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} BadBot2 [NC]
RewriteRule ^(.*)$ - [F,L]

This seemed to be creating a loop reaching that 10 internal redirects limit.

So... I commented out my custom 403 page... and it stopped.

What do you think is the best solution here?

One of the suggestions from the support team was to use this:

BrowserMatchNoCase BadBot1 stopthebot
Deny from env=stopthebot

The above would not create that loop in conjunction with my custom 403 page.

What is the best option for if we want to continue using our custom 403 and stop bots/user agents of our choice?

Thank you

[keyplyr]

Don't use this:

RewriteRule ^(.*)$ - [F,L]

To continue using your custom 403 page without the infinite loop you need to allow it in your RewriteRule... Use this:

RewriteRule !^custom403page\.html$ - [F]

Note: edit to match exact name of your custom 403 page.

[lucy24]

I do it this way. At the very beginning of all RewriteRules--before the ones with [F] flags--say (substituting the exact URL of your own 403 document);

RewriteRule ^forbidden\.html - [L]

Every mod that issues 403s needs a separate hole poked for it. The above version is for mod_rewrite. If you also use mod_authzthingummy, possibly in conjunction with mod_setenvif, then that exemption looks like this:

<Files "forbidden.html">
Order Deny,Allow
Allow from all
</Files>

Note that here you use only the filename, not the full URL path as in mod_rewrite. Apache 2.4 will have different content, involving <Require>, but the same <Files> envelope.

The mention of a support team implies that you are on shared hosting. If so, there is probably a default name for standard ErrorDocuments: missing.html, forbidden.html and so on. The Files exemption is then already present in the config file--but it does no harm to say it over again in htaccess.

Incidentally:
#1 when you are not capturing for reuse, there is no need for parentheses (.*) in the body of the rule. You can say either ^ alone or .? alone.
#2 The [F] flag implies [L]. (So does the [G] flag.) The supplementary [L] will do no harm, but is not needed.
#3 Don't use NoCase (or the [NC] flag) unless you really need to allow for all possible casings. It's trivial to type--like checking the "ignore case" box in a text editor--but it means the server has to do twice as much work.

[smallcompany]

Thanks very much.

So I opted for RewriteRule to my custom 403 page. And I left the ErrorDocument 403 /error403.php active in .htaccess.

Thank you