Welcome to WebmasterWorld Guest from 54.196.2.131

Forum Moderators: Ocean10000 & incrediBILL & phranque

RewriteCond vs. BrowserMatchNoCase

options to block bad bots

     
6:01 am on Oct 12, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 2, 2006
posts:2204
votes: 6


Hi,

Errors in Apache logs:

AH00124: Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.

After tracing it back, it looked like "bad bots" being redirected to a custom 403 page were causing this. Server support confirmed it, and I found references about similar cases. That time related config in .htaccess was:
ErrorDocument 403 /error403.php
RewriteCond %{HTTP_USER_AGENT} BadBot1 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} BadBot2 [NC]
RewriteRule ^(.*)$ - [F,L]

This seemed to be creating a loop reaching that 10 internal redirects limit.

So... I commented out my custom 403 page... and it stopped.

What do you think is the best solution here?

One of the suggestions from the support team was to use this:
BrowserMatchNoCase BadBot1 stopthebot
Deny from env=stopthebot

The above would not create that loop in conjunction with my custom 403 page.

What is the best option for if we want to continue using our custom 403 and stop bots/user agents of our choice?

Thank you
6:07 am on Oct 12, 2017 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:10116
votes: 550


Don't use this:
RewriteRule ^(.*)$ - [F,L]


To continue using your custom 403 page without the infinite loop you need to allow it in your RewriteRule... Use this:

RewriteRule !^custom403page\.html$ - [F]
Note: edit to match exact name of your custom 403 page.
6:30 pm on Oct 12, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:14259
votes: 552


I do it this way. At the very beginning of all RewriteRules--before the ones with [F] flags--say (substituting the exact URL of your own 403 document);
RewriteRule ^forbidden\.html - [L]
Every mod that issues 403s needs a separate hole poked for it. The above version is for mod_rewrite. If you also use mod_authzthingummy, possibly in conjunction with mod_setenvif, then that exemption looks like this:
<Files "forbidden.html">
Order Deny,Allow
Allow from all
</Files>
Note that here you use only the filename, not the full URL path as in mod_rewrite. Apache 2.4 will have different content, involving <Require>, but the same <Files> envelope.

The mention of a support team implies that you are on shared hosting. If so, there is probably a default name for standard ErrorDocuments: missing.html, forbidden.html and so on. The Files exemption is then already present in the config file--but it does no harm to say it over again in htaccess.

Incidentally:
#1 when you are not capturing for reuse, there is no need for parentheses (.*) in the body of the rule. You can say either ^ alone or .? alone.
#2 The [F] flag implies [L]. (So does the [G] flag.) The supplementary [L] will do no harm, but is not needed.
#3 Don't use NoCase (or the [NC] flag) unless you really need to allow for all possible casings. It's trivial to type--like checking the "ignore case" box in a text editor--but it means the server has to do twice as much work.
9:54 pm on Oct 23, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 2, 2006
posts:2204
votes: 6


Thanks very much.

So I opted for RewriteRule to my custom 403 page. And I left the ErrorDocument 403 /error403.php active in .htaccess.

Thank you