Help needed for rDNSbot plus block IP,s

[whatsdoin]

Gang I need help...I recently followed a link from Incredibills.blogspot,and found a great couple of threads talking about a subject,which had aroused my suspicions in the hosting back office..regarding site hijacking and fake robots and scrapers.
Thanks to the guys of those threads i organised my .htaccess files and.. surprise my my real traffic and views have been steadily on the rise.

However guys after putting in this code below in .htaccess. from those particular threads..
It works great..but for some reasons i cannot block individual IP's.
The ip,s I want to block and put in .htacces still continue to crawl my site..what can i do to rectify this?

many thanks

here is exactley my .htaccess
=========================================
AddHandler application/x-httpd-php5 .php
<Files 403.shtml>
order allow,deny
allow from all
</Files>
<FilesMatch "\.(s?html?¦php[45]?)$">
#
BrowserMatchNoCase Googlebot rDNSbot
BrowserMatchNoCase msnbot rDNSbot
BrowserMatchNoCase Slurp rDNSbot
BrowserMatchNoCase Teoma rDNSbot
#
SetEnvIf Request_URI "/path-to-custom-403-page\.shtml$" AllowAll
#
Order Deny,Allow
Deny from env=rDNSbot
Allow from env=AllowAll
Allow from googlebot.com
Allow from search.live.com
Allow from crawl.yahoo.net
Allow from ask.com
Allow from inktomisearch.com

</FilesMatch>
deny from 38.99.44.99
deny from 72.36.94.152
deny from 128.174.254.29
deny from 216.95.221.39
deny from 213.232.196.107
deny from 81.172.95.166
deny from 66.249.17.251
============================================

[wilderness]

Unfortunately and in this instance, you cannot have the best of two worlds (options)

deny, allow
[httpd.apache.org...]
or
allow, deny
[askapache.com...]

some more reading
[webmasterworld.com...]
[webreference.com...]

When implementing "white-listing" and the deny,allow option, it becomes necessary to deny IP ranges with Rewrite.

RewriteEngine on (if not already on)
RewriteCond %{REMOTE_ADDR} ^38\.99\.44\.99 [OR]
RewriteCond %{REMOTE_ADDR} ^72\.36\.94\.152
RewriteRule .* - [F]

Please note; Your going to learn very fast that limiting your denies to a solitary Class D range will come back very quickly to bite you in the backside.
A more effective action is to deny with the following options:
1) a range in the Class D
2) The entire Class D
3) The entire range of the provider your denying.

[jdMorgan]

wilderness is correct, you cannot have two Order directives in an .htaccess file, unless they are within containers ( such as <Files> or <Limit> ) that make them mutually-exclusive. If two Order directives are not mutually-exclusive, then only the last one found will be used.

However, the problem with your code is likely that the second Order directive is scoped only for the files specified in the <FilesMatch> container, and the server is therefore probably defaulting to Deny Allow with an Allow from all inherited from the server configuration file.

This can be fixed by moving the <FilesMatch> opening and closing tags, along with a few unrelated tweaks:

AddHandler application/x-httpd-php5 .php
#
# Remove the following four lines -- They are redundant. Put the
# URL-path of your 403.shtml page into the bolded line below
# <Files 403.shtml>
# order allow,deny
# allow from all
# </Files>
#
# If my guess shown here is incorrect, put the correct
# local URL-path to your custom 403 page into this line
[b]SetEnvIf Request_URI "/403\.shtml$" AllowAll[/b]
#
Order Deny,Allow
Allow from env=AllowAll
#
<FilesMatch "\.(s?html?¦php[45]?)$">
#
BrowserMatchNoCase Googlebot rDNSbot
BrowserMatchNoCase msnbot rDNSbot
BrowserMatchNoCase Slurp rDNSbot
BrowserMatchNoCase Teoma rDNSbot
#
Deny from env=rDNSbot
Allow from googlebot.com
Allow from search.live.com
Allow from crawl.yahoo.net
Allow from ask.com
Allow from inktomisearch.com
#
</FilesMatch>
#
Deny from 38.0.0.0/8
Allow from 38.114.104.0/24
Deny from 72.36.94.152
Deny from 128.174.254.29
Deny from 216.95.221.0/24
Deny from 213.232.196.0/24
Deny from 81.172.95.166
Deny from 66.249.0.0/19

I should note that the <FilesMatch> container is not required. Its purpose is to prevent the rDNS checks from being done on every single HTTP request to your server. These rDNS checks are expensive in terms of server performance, so only the filetypes listed in the <FilesMatch> are checked. This approach is based on the presumption that if they can't fetch your pages, then they can't find your images, CSS, etc.

When I posted the original code, I had no expectation of making a one-size-fits-all, cut-n-paste solution, so it's likely that everyone will need to tweak the code to suit their site.

Replace all broken pipe "¦" characters above with solid pipe characters before use; Posting on this forum modifies the pipe characters.

Jim

[whatsdoin]

Man you guys are the BOMB..thanks for the help and taking time out in to answer my questions...i Appreciate that..and hopefully can extend my learnings to others in life as you have when the time arises.
my best.
p.s
Thanks JD yours was the code that helped me most,which you correctly picked i cut and pasted :)

[edited by: whatsdoin at 11:57 pm (utc) on Aug. 2, 2007]

[whatsdoin]

Just as a follow on guys "Live search" bot seems to be ok with the script but it seems to block MSN bot itself....when i check if it is not spoof bot it is actually msn..is this normal?

[jdMorgan]

No, not normal. But msnbot may be doing something different from when I last looked at its rDNS -- Update the code to add or change the msnbot reverse-DNS hostname to match whatever they're using today. (Please post and let us know, too)

Jim

[whatsdoin]

Thanks guys ..i must state it is all chinese to me but i have tried this..and still no go...I think you guys are right canot have best of both worlds...particualrly 213.232.196.107 is giving me the shats from russia...no agent no bot nothing
================
AddHandler application/x-httpd-php5 .php
SetEnvIf Request_URI "/path-to-custom-403-page\.shtml$" AllowAll
#
Order Deny,Allow
Allow from env=AllowAll
#
<FilesMatch "\.(s?html?¦php[45]?)$">
#
BrowserMatchNoCase Googlebot rDNSbot
BrowserMatchNoCase msnbot rDNSbot
BrowserMatchNoCase Slurp rDNSbot
BrowserMatchNoCase Teoma rDNSbot
#
Order Deny,Allow
Deny from env=rDNSbot
Allow from googlebot.com
Allow from search.live.com
Allow from crawl.yahoo.net
Allow from ask.com
Allow from inktomisearch.com
#
</FilesMatch>
#
deny from 38.99.44.99
deny from 128.174.254.29
deny from 216.95.221.39
deny from 213.232.196.107
deny from 81.172.95.166
deny from 66.249.17.251
deny from 72.36.94.152
deny from 209.128.83.201

=============================

[wilderness]

I use Rewrite with white-listing (as previously provided) and it functions.

Suggest you replace the followin two lines
deny from 38.99.44.99
deny from 213.232.196.107

with the following:

RewriteEngine on (if not already on)
RewriteCond %{REMOTE_ADDR} ^38\. [OR]
RewriteCond %{REMOTE_ADDR} ^213\.232\.196\.
RewriteRule .* - [F]

[whatsdoin]

wilderness & jdMorgan..thanks very much for your help.
JD asked me to keep all informed..so here goes.I tinkered around a little..

JD as stated before Live search is ok with the set up but Msn bot is not..i will give you copy here down below to look.

This is my current .htaccess,and thanks to you guys so far so good I hope it also helps someone else..it has even blocked 213.232.196.107..yipeee
I will show you in a minute.
==============================================
AddHandler application/x-httpd-php5 .php
#
RewriteEngine on
#
SetEnvIf Request_URI "/path-to-custom-403-page\.shtml$" AllowAll
#
Order Deny,Allow
Allow from env=AllowAll
#
<FilesMatch "\.(s?html?¦php[45]?)$">
#
BrowserMatchNoCase Googlebot rDNSbot
BrowserMatchNoCase msnbot rDNSbot
BrowserMatchNoCase Slurp rDNSbot
BrowserMatchNoCase Teoma rDNSbot
#
Order Deny,Allow
Deny from env=rDNSbot
Allow from googlebot.com
Allow from search.live.com
Allow from crawl.yahoo.net
Allow from ask.com
Allow from inktomisearch.com
#
</FilesMatch>
#
RewriteCond %{REMOTE_ADDR} ^38\. [OR]
RewriteCond %{REMOTE_ADDR} ^213\.232\.196\.
deny from 128.174.254.29
deny from 216.95.221.39
deny from 81.172.95.166
deny from 66.249.17.251
deny from 72.36.94.152
deny from 209.128.83.201
#
RewriteRule .* - [F]
======================================

-script above blocking 213.232.196.107

Host: 213.232.196.107 /permalink.php?article=**********.txt
Http Code: 403 Date: Aug 05 03:33:07 Http Version: HTTP/1.1 Size in Bytes: -
Referer: -
Agent: -

/permalink.php?article=*****.txt
Http Code: 403 Date: Aug 05 04:44:05 Http Version: HTTP/1.1 Size in Bytes: -
Referer: -
Agent: -

/permalink.php?article=*******.txt
Http Code: 403 Date: Aug 05 10:06:02 Http Version: HTTP/1.1 Size in Bytes: -
Referer: -
Agent: -

------------------------------

Live search =

livebot-65-55-208-184.search.live.com /permalink.php?article=*********.txt
Http Code: 200 Date: Aug 05 15:22:38 Http Version: HTTP/1.0 Size in Bytes: 40367
Referer: -
Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)

/permalink.php?article=******.txt
Http Code: 200 Date: Aug 05 15:22:38 Http Version: HTTP/1.0 Size in Bytes: 33187
Referer: -
Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)

/permalink.php?article=******.txt
Http Code: 200 Date: Aug 05 15:22:39 Http Version: HTTP/1.0 Size in Bytes: 40304
Referer: -
Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)
----------------------------------

And now this..i have checked the ip and it of course belongs to MSN..so Live search is OK but this is not.?

65.54.188.90
Http Code: 403 Date: Aug 05 07:46:50 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)

Http Code: 403 Date: Aug 05 10:37:10 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)

Http Code: 403 Date: Aug 05 14:49:46 Http Version: HTTP/1.0 Size in Bytes: -
Referer: -
Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)
----------------------
once again thank you Both..I will monitor to see,and if all OK after a few days with the htaccess + rewrite let you guys know...but MSN i dont know whats happening.

[wilderness]

RewriteCond %{REMOTE_ADDR} ^38\. [OR]
RewriteCond %{REMOTE_ADDR} ^213\.232\.196\.
deny from 128.174.254.29
deny from 216.95.221.39
deny from 81.172.95.166
deny from 66.249.17.251
deny from 72.36.94.152
deny from 209.128.83.201
#
RewriteRule .* - [F]

the above is NOT correct.

The lines I suggested go into an entirely different section of your htaccess.
1) after all deny froms
2) after close mod lines EX:
a) allow from all
deny from env=keep_out
</Limit>

3) then add
RewriteEngine on
RewriteCond %{REMOTE_ADDR} ^38\. [OR]
RewriteCond %{REMOTE_ADDR} ^213\.232\.196\.
RewriteRule .* - [F]

[wilderness]

In addition, each time you make modifications to your htaccess , it's a good idea to visit your website (s) to confirm that you have not initiated a syntax error (which you have) that will prevent you site (s) from functioning and providing all visitors with a 500 error.

[wilderness]

I use Rewrite with white-listing (as previously provided) and it functions.

Suggest you replace the followin two lines
deny from 38.99.44.99
deny from 213.232.196.107

with the following:

RewriteEngine on (if not already on)
RewriteCond %{REMOTE_ADDR} ^38\. [OR]
RewriteCond %{REMOTE_ADDR} ^213\.232\.196\.
RewriteRule .* - [F]

whatsdoing,
My apologies for the confusion.
Your misunderstanding of the addition is due my use of the word replace even though the intent of replace is the same, the replace is done in a different section of the file.

Don

[whatsdoin]

Mate please dont apologise..it should me that apologises for the nob questions..thanks for putting up with me.

I will now try what you said.
You are correct about the 500's just one slip in the formula seems to trigger it off.

[Key_Master]

65.54.188.90 reverse resolves to msnbot.msn.com

:):):):):)

[jdMorgan]

Thanks Key_Master,

OK, so that makes the Allow list:

Allow from googlebot.com
Allow from search.live.com
Allow from msnbot.msn.com
Allow from crawl.yahoo.net
Allow from ask.com
Allow from inktomisearch.com

Jim