exceeding redirect limits with Rewrite commands

htaccess rewrite commands deny me access

[revrob]

I am very new to php and htaccess.

I am using bot traps to deal with unwanted visitors particularly those looking for vulnerable files that no longer exist on my site (an older version of WebCalendar that WAS indexed on google and got hacked for an IRC relay).

I am catchng the crawling robots that ignore robots.txt with bot traps.

But to catch the "direct" hits I am trying to redirect bots which are looking directly (not crawling) for the vulnerable files based on their existing databases of my site.

The bots are looking for files in /WebCalendar/ and sub directories. They don't crawl for them, they already know about them and make a single visit to access them.

I am having trouble with configuring the .htaccess file especially the results of some of the Rewrite commands.

In particular the command

# RewriteRule ^(.*)$ /WebCalendar/trap.php [L]
crashes the site and I get these warnings in my error file when I test the traps - and the site crashes even if I don't test them - just putting that statement in and trying to access my site normally doesn't work.

[Thu Sep 27 17:05:24 2007] [error] [client my iP] mod_rewrite: maximum number of internal redirects reached. Assuming configuration error. Use 'RewriteOptions MaxRedirects' to increase the limit if neccessary.
[Thu Sep 27 17:05:24 2007] [error] [client my iP] Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
[Thu Sep 27 17:19:05 2007] [alert] [client my iP] /var/www/vhosts/mydomain/httpdocs/.htaccess: RewriteCond: bad argument line '%{REQUEST_URI}!^WebCalendar/trap.php$'

My .htaccess file reads as follows: (I've remarked the whole Rewrite section in question because of the problem)

Rewriteengine ON
RewriteRule ^$ /index.html [R,NC,L]

# RewriteCond %{REQUEST_URI} ^WebCalendar/
# RewriteCond %{REQUEST_URI}!^WebCalendar/trap.php$
# RewriteCond %{REQUEST_URI}!^WebCalendar/finalwarning.php$
# should rewrite everything starting with WebCalendar/
# except the trap.php page and finalwarning.php
# RewriteRule ^(.*)$ /WebCalendar/trap.php [L]
# RewriteRule ^(.*)$ /trap/trap.php [L]
# should send everything through this script.

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm
<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "\.php$">
order allow,deny
allow from all
# </FilesMatch>

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm

#<Files .htaccess>
#order allow,deny
#deny from all
#</Files>

#order allow,deny
deny from # IP address list
#allow from all

</FilesMatch>

Many thanks for the assistance. Obviously I need to deal with those Redirect limits but I don't know how to.

[revrob]

I got this refined a bit to the point where it no longer crashes the site (needed a space inserting in one of the Rewrite commands) but it doesn't redirect as it is meant to. The traps work if accessed directly by file name, but they don't fire off if someone goes to the /WebCalendar/ directory.

Rewriteengine ON
RewriteRule ^$ /index.html [R,NC,L]

RewriteCond %{REQUEST_URI} ^WebCalendar/
RewriteCond %{REQUEST_URI}!^trap/warning.php$
RewriteCond %{REQUEST_URI}!^trap/trap.php$
# should rewrite everything starting with WebCalendar/
# except the warning.php
RewriteRule ^(.*)$ /trap/get_lost.php [L]
# should send everything through this script.

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm
<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "\.php$">
order allow,deny
allow from all
# </FilesMatch>

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm

#<Files .htaccess>
#order allow,deny
#deny from all
#</Files>

#order allow,deny
deny from #list of IPs
#allow from all

</FilesMatch>

[g1smd]

What happens if you trade in REQUEST_URI for THE_REQUEST in the three conditions?

That is what I would use, but I can't explain why. It's just a hunch.

[jdMorgan]

I could not tell for certain what directory you've got your trap scripts in -- i.e. whether or not they are in or below "Webcalendar," so I left those patterns un-anchored.

Because you are rewriting to /trap/get_lost.php, I assume that script is at example.com/trap/get_lost.php

The redirection looping problem is prevented by specifically excluding /trap/get_lost.php from being rewritten (again and again) to itself.

There are some inconsistencies here that I can't resolve from the posted info, but perhaps this will get you closer -- Note the leading slashes on REQUEST_URI patterns, changes to flags, etc.

RewriteEngine on
RewriteRule ^$ /index.html [R=301,L]
#
RewriteCond %{REQUEST_URI} !/trap/warning\.php$
RewriteCond %{REQUEST_URI} !/trap/trap\.php$
RewriteCond %{REQUEST_URI} !^/trap/get_lost\.php$
# should rewrite everything starting with WebCalendar/ except the warning.php
RewriteRule ^WebCalendar/ /trap/get_lost.php [L]
# should send everything through this script. 

There remain several opportunities for efficiency improvements, but again, these depend on the specific URL-paths to the scripts.

Jim

[revrob]

You are a star! That works beautifully. The "inconsistencies" were due to me munging some of the file names in my .htaccess fragment but then leaving others unchanged

The fragment is now:

Rewriteengine ON
RewriteRule ^$ /index.html [R,NC,L]

#
RewriteCond %{REQUEST_URI}!/trap/your_last_warning\.php$
RewriteCond %{REQUEST_URI}!^/trap/get_lost\.php$
# should rewrite everything starting with WebCalendar/ except the warning.php
RewriteRule ^WebCalendar/ /trap/get_lost.php [L]
# should send everything through this script.

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm
<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "\.php$">
order allow,deny
allow from all

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm

deny from # list of IPs

</FilesMatch>

The folder being redirected is /WebCalendar/
The warning file is /trap/your_last_warning.php
The trap script is /trap/get_lost.php

The result is that a request for something like
"/WebCalendar/anyoldrubbish.whateverfiletype" now fires off the trap, sends me an email, writes the IP to .htaccess and denies access, while displaying a message for an innocent inheritor of a blocked IP to contact the webmaster.
On further attempts to access the site home page they get my customised error page which also has contact details.

This is exactly what I wanted. Thanks to those both here and in the php forum for helping me get it right! That will keep the bots chasing old WebCalendar vulnerabilities off my site for a short while!

[g1smd]

Can you provide a link here to the other discussion and vice versa, as your solution might be useful to other people too...

[revrob]

Here is the discussion on php serverside scripting group - sorry I'm not familiar with how this board's software works so this is just text.
[webmasterworld.com...]

There is a thread here which predates this discussion and got me interested - but I didn't participate
[webmasterworld.com...]

All is working well now except for one minor glitch in the php script which doesn't report the client IP in the email sent to webmaster. I'm following that up in php scripting forum.

[revrob]

Well I THOUGHT everything was working fine but it isn't. Some really weird stuff going on.

This morning I couldn't access my php files. I discovered eventually that if I remarked a single IP in my deny from list in the .htaccess file, that situation resolved itself. The IP is 70.86.152.xx <snip>

Merely remarking that "deny from 70.86.152.xx" line in my .htaccess file allows me back into my php files. How on earth does that work?

My .htaccess file is behaving very weirdly - if I ban myself by adding my own IP to the file, I now find that I am banned from ordinary html files, but can still access many of my php applications by using a url direct to the file such as /gallery/index.php or phpBB/index.php - the application will open with a slightly strange display but it still works.

I have tested the current .htaccess file as follows:

include the IP 70.86.152.xx in the deny list - bans me from accessing all my php pages. Remark that entry - all my php pages become usable again. This happens even if my own IP is NOT on the ban list.

If I remark that entry and use my bot-trap to deny myself - my IP is appended to the bottom of .htaccess, I receive the email notifying webmaster of the banning. the banning has the following results www.mydomain/ - Apache test page
www.example.com/index.html - My own 403 error page
www.example.com/phpBB/ - standard 403 error page
www.example.com/phpBB/index.php - my forum main page (minus graphics) - and it works

Similarly for my other php based pages - depending on how the request is made - I can gain access even when banned.
Obviously I am not happy at the thought that a banned IP can still access my php files.

This seems really weird. Any ideas? I've tried remarking out various things in .htaccess and can't seem to solve it but obviously it is something there that is doing it. I also can't understand how including the mysterious IP of another server can affect my access to php files.

.htaccess currently looks like this
****************************************
Rewriteengine ON
RewriteRule ^$ /index.html [R,NC,L]

RewriteCond %{REQUEST_URI}!/trap/your_last_warning\.php$
RewriteCond %{REQUEST_URI}!^/trap/get_lost\.php$
# should rewrite everything starting with WebCalendar/ except the warning.php
RewriteRule ^WebCalendar/ /trap/get_lost.php [L]
# should send everything through this script.

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm
<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "\.php$">
order allow,deny
allow from all

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm

deny from # list of IPs
# deny from 70.86.152.xx - this is the weird IP I have to remark out
</FilesMatch>
****************************************

Any ideas anyone - this has got me totally stumped. I'm brand new to Apache and php so please answer in longhand!

Error log entry if I try and access a php page with that IP in my deny from list
*************************************
[Sat Sep 29 14:58:32 2007] [error] [client MYIP.***.***.**] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/phpBB/index.php, referer: http://www.example.com/index.html
[Sat Sep 29 14:58:32 2007] [error] [client MYIP.***.***.**] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/phpBB/index.php, referer: http://www.example.com/index.html
****************************************

So my questions are:

Why does remarking/not remarking that IP make a difference to whether I can access php files (when not banned in .htaccess)
Why, when I ban myself, can I still access the php files direct?

All help gratefully received.

[edited by: jatar_k at 2:11 pm (utc) on Sep. 29, 2007]

[edited by: jdMorgan at 2:38 pm (utc) on Sep. 29, 2007]
[edit reason] no specific ips thanks [/edit]

[jdMorgan]

You'll need to look at the file after the script adds your IP address to the ban list. Does it add your IP address outside the <FilesMatch> container? If so, then you won't be getting the results you expect, and the script will likely need to be modified.

Jim

[revrob]

Yes it does but I haven't found (up till now), that in practice that it makes any difference whether the deny from statement is before or after the <FilesMatch> container. However I'm open to all suggestions as to how to make my script put it inside the container.

At the moment (and things don't seem to be all that consistent at present)

with a deny statement for my own IP inside the container, I can connect to my site index page. If I try and connect to an index.php page in my forum I get a custom 403. If I try and connect to the forum directory with no page specified I get the same custom 403.

That means a banned IP can visit the site but not access any php files which seems fine to me, and kinder to the innocent victims who get banned by inheriting a banned IP from someone else. I'll definitely cope with that.

If I put the deny from statement for my own IP outside the container, where the bot trap script would put it, the following happens

my site index page -/index.html - custom 403 error page
my site root - Apache Test page
my forum root - standard 403 page
my forum index.php page - access to forum but without the graphics

this seems to be the worst case scenario - where banned IPs can get to the php stuff but not to the rest of the site!

So any advice as to how I can get the "deny from IP" statement inside the container?

the relevant section of the bot trap is:

<?php
function userIP(){
switch ($_SERVER){
case 'HTTP_CLIENT_IP':
$userip = $_SERVER['HTTP_CLIENT_IP'];
break;
case 'HTTP_X_FORWARDED_FOR':
$userip = $_SERVER['HTTP_X_FORWARDED_FOR'];
break;
case 'HTTP_X_FORWARDED':
$userip = $_SERVER['HTTP_X_FORWARDED'];
break;
case 'HTTP_FORWARDED_FOR':
$userip = $_SERVER['HTTP_FORWARDED_FOR'];
break;
case 'HTTP_FORWARDED':
$userip = $_SERVER['HTTP_FORWARDED'];
break;
default:
$userip = $_SERVER['REMOTE_ADDR'];
break;
}
return $userip;
}

function tel_me(){
$day = date("Y-m-d-(D)-H:i:s",time());
$from = "badbots@mydomain\r\n"; //edit for the right email address
$to = "badbots@mydomain"; //edit for the right email address
$subject = "Alert: bad robot";
$msg = "A bad bot hit ". $_SERVER['REQUEST_URI'] ."\nat ". $day . " \n";
$msg .= "address is " . $bot_ip . "\nagent is " . $_SERVER['HTTP_USER_AGENT'] . "\n";
$msg = wordwrap($msg, 70);
mail($to, $subject, $msg, "From: $from");
}

function block_bot($t, $f){
$fh = fopen($f, 'ab');// open in binary mode just in case
fwrite($fh, $t);
fclose($fh);
}

$bot_ip = userIP();
// block the bot
$txt = "deny from $bot_ip\n";
$file = '/var/www/vhosts/mydomain/httpdocs/.htaccess'; //edit for path to your htaccess file
block_bot($txt, $file);
tel_me();

?>

Many thanks
I'm still mystified about how adding that particular 70...... IP to the deny list messes things up!

[jdMorgan]

Questions: Who set up this server -- you or a hosting company, and what version of Apache is it?

Jim

[revrob]

Thanks for the reply

Hosting company runs the Apache server. I know zilch about such things!

The standard 403 page says
Apache/2.0.52 (CentOS) Server at www.mydomain Port 80

Is there any other info I ought to be able to find that you want? I use a Plesk 8.2.0 interface for managing the domain.

[revrob]

---

System: The following message was spliced on to this thread from: http://www.webmasterworld.com/apache/3465772.htm [webmasterworld.com] by jdmorgan - 3:44 pm on Oct. 1, 2007 (CDT -5)

---

I've put this in a new thread - it has me and my domain host stumped.

This morning I couldn't access my php files. I discovered eventually that if I remarked/commented a single IP in my deny from list in the .htaccess file, that situation resolved itself. The IP is 70.86.152.xx <snip> - (Full IP addresses are not allowed in the forum)

Merely remarking that "deny from 70.86.152.xx" line in my .htaccess file allows me back into my php files. How on earth does that work?

My .htaccess file is behaving very weirdly - if I ban myself by adding my own IP to the file, I now find that I am banned from ordinary html files, but can still access many of my php applications by using a url direct to the file such as /gallery/index.php or phpBB/index.php - the application will open with a slightly strange display but it still works.

I have tested the current .htaccess file as follows:

include the IP 70.86.152.xx in the deny list - bans me from accessing all my php pages. Remark that entry - all my php pages become usable again. This happens even if my own IP is NOT on the ban list.

If I remark that entry and use my bot-trap to deny myself - my IP is appended to the bottom of .htaccess, I receive the email notifying webmaster of the banning. the banning has the following results www.mydomain/ - Apache test page
www.example.com/index.html - My own 403 error page
www.example.com/phpBB/ - standard 403 error page
www.example.com/phpBB/index.php - my forum main page (minus graphics) - and it works

Similarly for my other php based pages - depending on how the request is made - I can gain access even when banned.
Obviously I am not happy at the thought that a banned IP can still access my php files.

This seems really weird. Any ideas? I've tried remarking out various things in .htaccess and can't seem to solve it but obviously it is something there that is doing it. I also can't understand how including the mysterious IP of another server can affect my access to php files. I've even tried an .htaccess file with the "deny from" list cut right down to that IP only, and it still causes the problem - so it is nothing to do with the length of the list.

My php stuff includes a Lazarus guestbook, a phpBB forum, a WebCalendar 1.1.6, and a Gallery photo application.

.htaccess currently looks like this
****************************************
Rewriteengine ON
RewriteRule ^$ /index.html [R,NC,L]

RewriteCond %{REQUEST_URI}!/trap/your_last_warning\.php$
RewriteCond %{REQUEST_URI}!^/trap/get_lost\.php$
# should rewrite everything starting with WebCalendar/ except the warning.php
RewriteRule ^WebCalendar/ /trap/get_lost.php [L]
# should send everything through this script.

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm
<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "\.php$">
order allow,deny
allow from all

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm

deny from # list of IPs
# deny from 70.86.152.xx - this is the weird IP I have to remark out
</FilesMatch>
****************************************

Any ideas anyone - this has got me totally stumped. I'm brand new to Apache and php so please answer in longhand!

Error log entry if I try and access a php page with that IP in my deny from list
*************************************
[Sat Sep 29 14:58:32 2007] [error] [client MYIP.***.***.**] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/phpBB/index.php, referer: http://www.example.com/index.html
[Sat Sep 29 14:58:32 2007] [error] [client MYIP.***.***.**] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/phpBB/index.php, referer: http://www.example.com/index.html
****************************************

So my questions are:

Why does remarking/not remarking that IP make a difference to whether I can access php files (when not banned in .htaccess)
Why, when I ban myself, can I still access the php files direct?

All help gratefully received.