protecting access to custom error pages for banned IPs

error pages htaccess allow deny

[revrob]

I have implemented a bot trap on my site, linked to robots.txt and feeding banned IPs to my .htaccess using "deny from" statements. On first time of access the users get taken to a page with details of how to email the webmaster but if they navigate away from that they obvously can't return to the site.

I also have customised error pages, and I want banned IPs to be able to access the 403 forbidden page even if they are banned from the rest of the site, so that a human who has inherited a banned dynamic IP can see a message with an email option to contact the webmaster to get unbanned. I know there is a way to do this with .htaccess but am unfamiliar with the syntax.

This is my .htaccess at present. The customised error pages work but not for the banned IPs.

I would like to know where to put in an allow statement for the 403 forbidden access page which will allow it for everyone, including the banned IPs in the deny list - and also what that statement should be.

Many thanks.

Current .htaccess file contents
*****************************
<Files .htaccess>
order allow,deny
deny from all
</Files>

ErrorDocument 403 /errors/403.html
ErrorDocument 404 /errors/404.html
ErrorDocument 500 /errors/500.html

order allow,deny
deny from ***.***.***.*** # list of banned IPs

allow from all
********************************

Many thanks.

[jdMorgan]

The key is use "Deny,Allow" which enables Allow directives to override Deny directives. Then we use mod_setenvif to set a couple of server variables, and Allow or Deny based on them using the "Allow from env=var" syntax supported by mod_access.

ErrorDocument 403 /errors/403.html
ErrorDocument 404 /errors/404.html
ErrorDocument 500 /errors/500.html
 
SetEnvIf Request_URI "(.htaccess¦.htpasswd)$" blockit
SetEnvIf Referer "iaea\.org$" blockit
SetEnvIf Request_URI "(403\.html¦robots\.txt)$" allowit
<Files *>
Order [b]Deny,Allow[/b]
Deny from env=blockit
Allow from env=allowit
 
# list of banned IPs
Deny from 123.45.67.89
Deny from 234.10.11
Deny from 38.0.0.0/24
Deny from 212.0.0.0/255.128.0.0
</Files>

Note that HTTP access to <anything>.htaccess and <anything>.htpasswd is always denied. This <anything> bit allows you to move the files around at will, or even to have different custom error pages in different directories.

HTTP access to <anything>403.html and <anything>robots.txt is always allowed. Allowing access to the 403 page is required to prevent an infinite loop when a user-agent is denied, and remember that some robots may treat any error on a robots.txt fetch as permission to index the entire site. So it's important to allow any user-agent to fetch robots.txt, even if it might be a bad-bot.

IP addresses are for example only; They're all just "made-up" numbers showing the various supported formats.

Change all broken pipe "¦" characters above to solid pipes before use; Posting on this forum modifies the pipe characters.

Jim

[revrob]

Thanks for the reply - I tried that, edited htaccess including the pipes, banned myself and tested - but I'm afraid I still got the standard server error pages and not my own. How do I trouble shoot this please? Sorry I'm newbie to php.

error files are /errors/403.html - 404.html - 500.html

robots.txt (details altered)
User-agent: *
Disallow: /R****/
Disallow: /*****.5old/
Disallow: /***********/
Disallow: /**********/
Disallow: /*********/
Disallow: /trapfile.php
Disallow: /trapfile2.php
Disallow: /trapscript-script.php
Disallow: /trap/
Disallow: /guestbook/
Disallow: /photo-gallery/
Disallow: /forumfolder/

the trapscript entry is
<?php
// author: seven-3-five, 2006-09-04, seven-3-five.blogspot.com
//this script is the meat and potatoes of the bot-trap
// 1. It sends you an email when the page /badbots.php is visited.
//The email contains various info about the visitor.
//2. It adds the directive
//'deny from $ip' ($ip being the visitor's ip address)
//to the bottom of your .htaccess file.

// SERVER VARIABLES USED TO IDENTIFY THE OFFENDING BOT

$ip = $_SERVER['REMOTE_ADDR'];
$agent = $_SERVER['HTTP_USER_AGENT'];
$request = $_SERVER['REQUEST_URI'];
$referer = $_SERVER['HTTP_REFERER'];

// CONSTRUCT THE EMAIL MESSAGE

$subject = 'bad-bots';
$email = '[e-mail]'; //edit accordingly
$to = $email;
$message ='ip: ' . $ip . "\r\n" .
'user-agent string: ' . $agent . "\r\n" .
'requested url: ' . $request . "\r\n" .
'referer: ' . $referer . "\r\n"; // often is blank

$message = wordwrap($message, 70);

$headers = 'From: ' . $email . "\r\n" .
'Reply-To: ' . $email . "\r\n" .
'X-Mailer PHP/' . phpversion();

// SEND THE MESSAGE

mail($to, $subject, $message, $headers);

// ADD 'deny from $ip' TO THE BOTTOM OF YOUR MAIN .htaccess FILE

$text = 'deny from ' . $ip . "\n";
$file = '.htaccess';

add_badbot($text, $file);

// Function add_bad_bot($text, $file_name): appends $text to $file_name
// make sure PHP has permission to write to $file_name

function add_badbot($text, $file_name) {
$handle = fopen($file_name, 'a');
fwrite($handle, $text);
fclose($handle);
}

?>

[wilderness]

Thanks for the reply - I tried that, edited htaccess including the pipes, banned myself and tested - but I'm afraid I still got the standard server error pages and not my own. How do I trouble shoot this please?

You need to view your error logs.

When you say "banned myself"?
Did you receive a 403-access denied error?
or
a 500 server failed error?

If the later,than you have an syntax error in your htacess.

If the first (403) than explore your IP and/or UA denies to your own browser or internet provider.

Don

[SeanW]

Put your error pages in a separate directory, ie /errors:

<Location /errors/ >
Order Allow,Deny
Deny from none
Allow from all
Satisfy Any
</Location>

That (satisfy any) should override any other commands that are preventing access to that dir.

Sean

[revrob]

To Sean W - my error files are already in a seperate directory - /errors/ - I've added the statement you suggest to my existing htaccess but it gives me a 500 error if I try and access my site so I've removed it again.

To Wilderness (sorry - can't see how to "quote" on my interface)

yes I get a 403 error.

With my existing htaccess file - it's like this:
If I go to the bot trap the first page I see is the one delivered by the script which is not an error page but a message page rendered by the script with info about contacting webmaster. So far so good.

If I refresh that (by now htaccess has been given my IP) I get a standard 403 (instead of my own 403)
"Forbidden - You don't have permission to access /noentry.php on this server. Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2.0.52 (CentOS) Server at www.mydomain Port 80"

If I try and access any other page on the site I get an "Apache 2 Test Page" which seems to be the way the domain host has set it up. That's better than nothing but I'd prefer to use own domain my customised error page.

With your suggested htaccess file - it's like this:
go to trap - get same page as delivered by the trap.
Refresh that - get the following
"Internal Server Error - The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator to inform of the time the error occurred and of anything you might have done that may have caused the error. More information about this error may be available in the server error log. Web Server at mydomain"

Go anywhere else on the site - get the 500 Internal Server Error.

The error log for the relevant section is:

[client myIP] PHP Notice: Undefined index: HTTP_REFERER in /var/www/vhosts/mydomain/httpdocs/trap-script.php on line 15
[Tue Sep 25 17:38:04 2007] [error] [client myIP] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/trap.php
[Tue Sep 25 17:38:04 2007] [error] [client myIP] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/errors/403.html
[Tue Sep 25 17:38:52 2007] [error] [client myIP] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/trap.php
[Tue Sep 25 17:38:52 2007] [error] [client myIP] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/errors/403.html
[Tue Sep 25 17:38:59 2007] [error] [client myIP] client denied by server configuration: /var/www/vhosts/mydomain/httpdocs/
[client 86.142.249.9] PHP Notice: Undefined index: HTTP_REFERER in /var/www/vhosts/mydomain/httpdocs/trap-script.php on line 15
[Tue Sep 25 17:44:46 2007] [alert] [client myIP] /var/www/vhosts/mydomain/httpdocs/.htaccess: </Files>deny> directive missing closing '>'
[Tue Sep 25 17:46:44 2007] [alert] [client myIP] /var/www/vhosts/mydomain/httpdocs/.htaccess: </Files>deny> directive missing closing '>'

Presumably that final error message is significant. I notice that although with my original htaccess file IPs added with a deny from statement, and appended after the final allow from all statement seem to work fine and deny the added IP while allowing the not banned ones, this version of htaccess can't cope with anything after the </Files> at the end. Unfortunately that is where my script is sending the IP's - to the end of the list. So it may be that the script I am using and the htaccess you are suggesting are not compatible.

Here's the htaccess file suggested here
************************************************************
ErrorDocument 403 /errors/403.html
ErrorDocument 404 /errors/404.html
ErrorDocument 500 /errors/500.html

SetEnvIf Request_URI "(.htaccess¦.htpasswd)$" blockit
SetEnvIf Referer "iaea\.org$" blockit
SetEnvIf Request_URI "(403\.html¦robots\.txt)$" allowit

<Files *>
Order Deny,Allow
Deny from env=blockit
Allow from env=allowit

# list of banned IPs
deny from ***.***.***.** #long list
</Files>deny from MYIP # my IP added here from the test exercise
*****************************************************************

Here's the previous htaccess file that works fine but doesn't deliver my own error files.

*************************************************************
<Files .htaccess>
order allow,deny
deny from all
</Files>

ErrorDocument 403 /errors/403.html
ErrorDocument 404 /errors/404.html
ErrorDocument 500 /errors/500.html

order allow,deny
deny from list of IPs

allow from all
**********************************************************

Here's the bot trap script again
**************************************************
<?php
// author: seven-3-five, 2006-09-04, seven-3-five.blogspot.com
//this script is the meat and potatoes of the bot-trap
// 1. It sends you an email when the page /badbots.php is visited.
//The email contains various info about the visitor.
//2. It adds the directive
//'deny from $ip' ($ip being the visitor's ip address)
//to the bottom of your .htaccess file.

// SERVER VARIABLES USED TO IDENTIFY THE OFFENDING BOT

$ip = $_SERVER['REMOTE_ADDR'];
$agent = $_SERVER['HTTP_USER_AGENT'];
$request = $_SERVER['REQUEST_URI'];
$referer = $_SERVER['HTTP_REFERER'];

// CONSTRUCT THE EMAIL MESSAGE

$subject = 'bad-bots';
$email = '[e-mail]'; //edit accordingly
$to = $email;
$message ='ip: ' . $ip . "\r\n" .
'user-agent string: ' . $agent . "\r\n" .
'requested url: ' . $request . "\r\n" .
'referer: ' . $referer . "\r\n"; // often is blank

$message = wordwrap($message, 70);

$headers = 'From: ' . $email . "\r\n" .
'Reply-To: ' . $email . "\r\n" .
'X-Mailer PHP/' . phpversion();

// SEND THE MESSAGE

mail($to, $subject, $message, $headers);

// ADD 'deny from $ip' TO THE BOTTOM OF YOUR MAIN .htaccess FILE

$text = 'deny from ' . $ip . "\n";
$file = '.htaccess';

add_badbot($text, $file);

// Function add_bad_bot($text, $file_name): appends $text to $file_name
// make sure PHP has permission to write to $file_name

function add_badbot($text, $file_name) {
$handle = fopen($file_name, 'a');
fwrite($handle, $text);
fclose($handle);
}

// start free emails for spider
$page = '';
for ( $i = 0; $i < 5000; $i++ )
{
$page .= new_email();
}

function new_email()
{
$email = '';
$letters_array = array('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r',
's', 't', 'u', 'v', 'w', 'x', 'y', 'z');
for ( $i = 0; $i < 17; $i++ )
{
$email .= ( $i!== 10 )? $letters_array[ mt_rand( 0, 25) ] : '@';
}
$email .= '.com';
$email = '<a href="mailto:' . $email . '">' . $email . "</a>\n";
return $email;

}

$page .= "Goodbye!";
echo $page;
?>
******************************

Thanks to all who are helping. Much appreciated.

[jdMorgan]

A major part of this question seems to be "why are my ErrorDocument directives being ignored?"

A possible cause is that "AllowOverride [httpd.apache.org] FileInfo" is not set in your server config.

Jim

[revrob]

In normal circumstances my error documents DO work - if I type a bad page url in the browser within my domain, the server does deliver my own customised 403 page. But not when I am banned by an IP entry in htaccess. Thanks for the reply.

[revrob]

Here's what I worked out with my domain host for the .htaccess file with my error documents in the root.
This allows my own customised error documents to be displayed to a user with an IP on the .htaccess ban list.

It also allows my bot trap to add banned IPs to the bottom of the file, they work as part of the .htaccess ban list but once a week I can manually move them into their proper place in the list.

.htaccess contents
*************************
Rewriteengine ON
RewriteRule ^$ /index.html [R,NC,L]

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm
<Files .htaccess>
order allow,deny
deny from all
</Files>

<FilesMatch "\.php$">
order allow,deny
allow from all

ErrorDocument 403 /403.htm
ErrorDocument 404 /404.htm
ErrorDocument 500 /500.htm

deny from ***.***.**.***
deny from ***.**.**.***

</FilesMatch>
***********************************

Thanks for the suggestions.