homepage Welcome to WebmasterWorld Guest from 54.161.200.144
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

This 122 message thread spans 5 pages: < < 122 ( 1 2 3 [4] 5 > >     
A Close to perfect .htaccess ban list - Part 2
adriaant




msg:1508343
 11:46 pm on May 14, 2003 (gmt 0)

<modnote>
continued from [webmasterworld.com...]



UGH, bad typo in my original post. Here's the better version (I wasn't able to re-edit the older post?):

I'm trying to ban sites by domain name, since there are recently lots of reference spammers.

I have, for example, the rule:

RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*stuff.*\.com/.*$ [NC]
RewriteRule ^.*$ - [F,L]

which should ban any sites containing the word "stuff"
www.stuff.com
www.whatkindofstuff.com
www.some-other-stuff.com

and so on.

However, it is not working, so I am sure I did not setup a proper pattern match rule. Anyone care to advise?

[edited by: jatar_k at 5:06 am (utc) on May 20, 2003]

 

Wizcrafts




msg:1508433
 7:35 pm on Sep 29, 2003 (gmt 0)

I'm running some wannabrowser tests right now and this is what I have just discovered:

First, I have two of filenames in question in my cgi-bin.
When I use Wannabrowser to GET FormMail.cgi, or php, or Form-Mail.cgi, I get my custom 403 page, but not banned. When I type the search for FormMail.pl, or formmail.pl, both get banned, because they are both traps and both file names exist on my server.

Second, and best of all, when I type in a search for a non-existent filename in the cgi-bin I also get a 403! I used cgi-bin/nonexistent.php as the filename. There is nothing in my .htaccess that I know of that should cause a 403 instead of a 404. I am going to go over the htaccess file word by word to try to find out why this is happening.

More to come...

More: I just checked the permission I had set on my cgi-bin and found them to be 751. I chmodded to 755 and I now get a 404 error for a file not found, instead of 403 forbidden.

Final test: formmail.php now gets banned as designed! The whole problem was a lack of read permission for the World group. I believe I set it at 751 to prevent outsiders from reading the scripts and stealing email addresses, etc (months ago). I have since learned to secure the individual files with 711 permissions, which work just fine.

Thanks to all who tried to help in this unusual situation.

Wiz

jdMorgan




msg:1508434
 7:51 pm on Sep 29, 2003 (gmt 0)

Wiz,

There is no need to have multiple copies of the trap, nor to use the Redirect directive (which causes a 302). Just "silently" rewrite *any* filename you want to trap (whether it actually exists or not) to the script:

RewriteRule (form.*mail¦mail.*(form¦to¦2)) /path_to_script.pl [NC,L]

Note that the above pattern is an unanchored, "compressed" version of that posted above. It also doesn't care what file type is requested, i.e. .cgi, .exe, .pl, etc. -- only that it contains formmail or mail2form, or anything similar to that. Also, note the stacked-group parentheses... Yes, you can! :)

Maybe using the above info to simplify your "trapping model" will get rid of whatever the problem is as a side-effect. Or... maybe not.

Jim

Wizcrafts




msg:1508435
 8:05 pm on Sep 29, 2003 (gmt 0)

Thanks for the compressed version Jim. It will replace the previous long code line in a few minutes. I have already disabled the redirectmatch line, now that the ban script is working.

There is one anomally I just discovered. After using Wannabrowser for the formmail tests I left it banned in .htaccess, then went and successfully fetched all the webpages I wanted. The only thing that was banned was access to files listed in the formmail condition. This means something else is overriding the "ban" environment set by the trap script and the directives in .htaccess. Would someone take a quick look at the Files restriction group and tell me if I need to rearrange any directives?

SetEnvIf Remote_Addr ^206\.194\.114\.2$ ban

SetEnvIf Request_URI ^(/includes/403\.html¦/robots\.txt)$ allowit

<Files *>
order deny,allow
allow from env=allowit
deny from env=ban
deny from 12.219.232.74
deny from 24.53.200.12
deny from 24.188.211.3
deny from 61.4.64.0/20
deny from 62.253.166.153
deny from 65.33.10.192
deny from 65.57.163.78
deny from 66.36.240.135
deny from 66.36.246.127
deny from 66.72.195.144
deny from 66.76.144.219
deny from 66.119.34.39
deny from 66.250.125.195
deny from 68.42.21.162
deny from 142.177.144.148
deny from 152.163.252.70
deny from 152.163.252.100
deny from 170.224.224.38
deny from 200.176.32.214
deny from 203.194.146.175
deny from 204.234.17.35
deny from 206.135.194.194
deny from 207.134.171.4
deny from 210.192.120.74
deny from 210.192.96.0/17
deny from 212.138.47.18
deny from 213.221.116.114
deny from 216.93.191.2
deny from 217.21.117.121
deny from 217.78.
deny from 220.73.25.68
deny from 220.73.165.
deny from 220.99.112.2
allow from all
</Files>

Wiz

Wizcrafts




msg:1508436
 12:28 am on Sep 30, 2003 (gmt 0)

Some technical problems are just too abstract to be understood by mere mortals.

I have been trying to get the env=ban portion of my .htaccess to actually ban something, to no avail...until just now.

By using the process of addition, I started with a minimal .htaccess in a test directory, with files and folders to match the real thing. Then, I added my banned IP list and the deny from allow from list. Everything worked as expected in the experimental directory, so I kept adding rules and conditions from my main .htaccess until I finally broke the experimental version. The code lines below are the ones that are causing my <Files *> directives to be ignored.

<Files *>
<LimitExcept GET POST>
deny from all
</LimitExcept>
</Files>

I will need a suitable substitute to insert into my rewrite conditions list, if anyone can help me with that. Or, possibly, the syntax is bad in this rule?

I should also mention that the ruleset above follows the main <Files *> allow-some, deny-some rules, abbreviated below.
<Files *>
order allow,deny
allow from all
allow from env=allowit
deny from env=ban
deny from 12.219.232.74
deny from 24.53.200.12
<snip>
</Files>

Wiz

claus




msg:1508437
 3:32 am on Sep 30, 2003 (gmt 0)

>> compressed version

These are the requests i got last time i checked (a couple of days ago):

/cgi-bin/mail.cgi
/cgi-bin/formmail.pl
/cgi-bin/Mail.pl
/cgi-bin/formmail.cgi

To catch "mail" as well, i believe you have to modify the compressed version a little bit, by adding a questionmark after "2)":

RewriteRule (form.*mail¦mail.*(form¦to¦2)?) /path_to_script.pl [NC,L]

Of course, if you have a legitimate file called "mail-something" (eg. "mail.html") that one will get caught too, which is not what you want, but the expression will need to become a little bit longer to take that into account also.

/claus

jdMorgan




msg:1508438
 4:12 am on Sep 30, 2003 (gmt 0)

Wiz,

You can replace that entire mod_access function with something like this using mod_rewrite:

# Restrict HTTP methods
RewriteCond %{REQUEST_METHOD} !^(GET¦HEAD¦OPTIONS¦POST)$
RewriteRule .* - [F]

claus,

Yes, everyone's preferences differ. Like the code above, I lifted that from my own files.

Jim

Wizcrafts




msg:1508439
 5:21 am on Sep 30, 2003 (gmt 0)

Thanks Jim and Claus.

I have to report that the compressed version of the form mail rule has caused chaos on my server! I have a page named fmsecurity.html and it all about FormMail security. As soon as I added the compressed line to my .htaccess it banned three visitors and myself who read my FormMail security page! Luckily I had my FTP client open and active when I got banned! Due to this unexpected result I have gone back to Balam's two line Form-Mail ruleset:

RewriteCond %{REQUEST_URI} (.?mail.?form¦form¦(GM)?form.?.?mail¦.?mail)(2¦to)?\.?(asp¦cgi¦exe¦php¦pl¦pm)?$ [NC]
RewriteRule .* path_to_hell.pl [F]

That code does what I want without banning my friendly visitors and myself.

Jim; thanks for the rewrite condition to replace the LimitExcept problem.

Wiz

jdMorgan




msg:1508440
 2:42 am on Oct 1, 2003 (gmt 0)

Wiz,

The problem you had with the compressed rule doesn't make any sense.

Running your "fmSecurity.html" request through a regex tester [regexlib.com] with the compressed rule produces a "No Match" result as expected, so you likely had some other problem such as a missing space or an extra (or missing) [OR] somewhere.

Jim

Wizcrafts




msg:1508441
 2:59 am on Oct 1, 2003 (gmt 0)

Jim;
The problem is not with the filename in this instance. It is caused by the page title; Security Alert For FormMail Script Users. Also, the word FormMail appears over and over in the text (including the first H1 heading). I was allowed to access the webpage and read it thoroughly, but once I left that page and followed a link to Wiz's Workshop I found myself banned as I reloaded that page. When I checked my weblog I saw that two other IPs were viewing fmsecurity with me and were also now on the banned list in .htaccess. I removed all three from the ban list and commented out the compressed code line, and reinstated the old code. When I went back to fmsecurity I was not banned again, and neither were the rest of the visitors going there all night.
I'm sorry, but the unanchored capture on the word form.?mail, without any extension, is triggered by the html inside the page. There is no other explanation, especially when the problem goes away by removing that codeline and replacing it with the one that looks for specific file extensions.

I think what I am trying to say is that the meta data in the header includes the word FormMail, thus it matched the "form" rule in the compressed rule. See header below:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
<title>Security Alert For FormMail Script Users</title>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="description" content="This is to alert users of Matt Wright's FormMail Perl script about the severe security issues that can arise from using older, unpatched versions of this script. We offer tips for securing, or replacing FormMail with a more modern, secure version.">
<meta name="keywords" content="cgi security, formmail security, prevent email relaying, formmail, nms mail script, hide recipient address in form, stop guestbook email harvesting">

Wiz

jdMorgan




msg:1508442
 4:10 am on Oct 1, 2003 (gmt 0)

Wiz,

> There is no other explanation

Well, there must be, because .htaccess processing is finished before any file is served, and a RewriteRule does not "examine" file content, it only looks at a derivative of the server variable {REQUEST_URI} (which is wny using a RewriteCond testing {REQUEST_URI} is usually only necessary for complex rewrites).

My guess is that there was some other problem, as stated above. Otherwise, I'd have banned myself at least a year ago. That code came off a working server, and was validated again today using the tool I cited above.

I'd be very interested if you ever identify the problem; Do you keep backups of previous .htaccess files by any chance?

Jim

Wizcrafts




msg:1508443
 4:17 am on Oct 1, 2003 (gmt 0)

Yeah, I have all kinds of backups. I'll investigate and let you know. It just seems to me that when I looked at the server log it said HTTP/1.1 and that would include fetching the head, title, description and keywords, would it not?

In the meantime I am happy with the uncompressed formmail ban ruleset. It goes by matching the filename and extension, whereas the compressed version doesn't care about extensions.

This time I'll test it in my experimental directory so as to not ban myself or others.

Wiz

Wizcrafts




msg:1508444
 5:20 pm on Oct 2, 2003 (gmt 0)

In my last post here I promised to report my findings regarding the difference in blocking between Claus and Balam's long RewriteRule-set and jbMorgan's compressed rules, for FormMail spammers and phishers.

My problem, unique to my website, was that only the long rule-set blocked requests for variations of the word Form and/or Mail scripts, while allowing html pages with the same names to pass through and be displayed (desired action). The compressed version also blocked requests for such pages of mine as formmailwarning.html, because it contains the Regexps "form" and "mail," and it does not check for filename extensions before blocking them.

I researched the problem and after much testing I have come up with a solution that causes the compressed version to behave the same as the long version.

[i]Balam's FormMail Request RewriteRules (157 bytes):[/i]
RewriteCond %{REQUEST_URI} (.?mail.?form¦form¦(GM)?form.?.?mail¦.?mail)(2¦to)?\.?(asp¦cgi¦exe¦php¦pl¦pm)?$ [NC]
RewriteRule .* /path_to/trap_script.pl [L]

[i]jbMorgan's compressed rules (75 bytes):[/i]
RewriteRule (form.*mail¦mail.*(form¦to¦2)) /path_to/trap_script.pl [NC,L]

[i]jbMorgan's compressed rules with Wizcrafts modification (121 bytes):[/i]
RewriteCond %{REQUEST_URI}!.*\.(html¦js¦css)
RewriteRule (form.*mail¦mail.*(form¦to¦2)) /path_to/trap_script.pl [NC,L]

If you carefully compare the three rule-sets you can see that
1) the first set blocks filenames by matching filenames with known dangerous file extensions.
2) The second one blocks by prefix match only, with no concern for what the extension is.
3) The third rule-set allows filenames with the desired extensions to pass, but blocks prefix matches on any other extensions.

While the? at the end of the 1st rule-set means that the extensions are optional, it works for me as desired nonetheless. The third, compressed rule-set accomplishes the same result with a savings of 36 bytes. If you do not have any filenames on your server that include "form" or "mail," you can leave off the RewriteCond line, as per rule-set #2, and it will block bad guys just fine, with a savings of an additional 46 bytes (82 bytes total saved).

As usual, anybody copying and pasting this code should replace the broken pipe characters with solid pipes from their own keyboard.

Submitted IMHO, after testing on my server with files that exist, and otherwise, by Wiz

Wizcrafts




msg:1508445
 5:56 pm on Oct 2, 2003 (gmt 0)

I need to correct one of my previous code snippets from my .htaccess file. Earlier I posted the following:

<Files *>
order allow,deny
allow from all
allow from env=allowit
deny from env=ban
deny from 12.219.232.74
deny from 24.53.200.12
<snip>
</Files>

However, after running a lot of tests I realized that the <Files> order is bad. The allowit rules were not taking effect at all. Here is how I corrected that problem:

<Files *>
order deny,allow
deny from env=ban
deny from 12.219.232.74
deny from 24.53.200.12
(snip)
allow from env=allowit
# allow from all # apparently not needed
</Files>

With this sequencing I am able to ban unwanted visitors, ie: SetEnvIf Remote_Addr ^216\.229\.194\.253$ ban, while allowing good guys, like myself or Wannabrowser to continue to access my website, even if I/they hit a banned script (on purpose). The allowit rule for Wannabrowser is: SetEnvIf Remote_Addr ^206\.194\.114\.2$ allowit. By placing "allow from env=allowit" at the bottom of the "deny from" rule-set, we can ensure that friends are not inadvertantly banned. Note that with this order (order deny,allow), the deny rules are processed first, then the allow rules kick in. Anything not specifically banned (or that is allowed by "allowit") is allowed through this rule-set; "Allowit" overrides "ban" for the same IP address.

Lastly, if you have a custom 403 page, and a robots.txt that you want the banned visitors to see, you have to add this allowit line: SetEnvIf Request_URI (/includes/403\.html¦/robots\.txt)$ allowit. Do NOT put a ^ in front of this code-line. It will prevent your cusom 403 from being accessed and cause an error message to appear in the generic 403 message that does display. That was my experience and removing the ^ fixed it.

Wiz

Note that this rule-set only applies to Files, not folders.

jdMorgan




msg:1508446
 6:17 pm on Oct 2, 2003 (gmt 0)

Wiz,

One neat thing about Order is that only the settings allow,deny or deny,allow are important. You can have your Deny from and Allow from directives in any order - They will be processed as dictated by the Order directive, not by the order they appear in. So, it is unnecessary to re-arrange your denies and allows just because you change your Order directive. This can make the code block easier to maintain and document, too.

Part of the confusion is caused by the Order name itself. It really indicates the precedence or priority between Allow and Deny directives and has nothing to do with the listing order of denies and allows.

Jim

Wizcrafts




msg:1508447
 2:18 am on Oct 3, 2003 (gmt 0)

it is unnecessary to re-arrange your denies and allows just because you change your Order directive.

Thanks for that info Jim. I will keep on listing things in logical order anyway, because I am used to doing things that way.

Oh, BTW, I added email notification to my trap script. It includes the IP address, the time banned, the filename and path request that triggered the script, the User Agent and the Method (GET or POST) used. I also tried to add an http_referer field but it causes 500 server errors when I include it in the Perl script, so it is out.

Wiz

jdMorgan




msg:1508448
 2:41 am on Oct 3, 2003 (gmt 0)

Wiz,

That's strange... Here's the syntax I'm using that writes a log file file including HTTP_REFERER. If you've got something different, maybe something like this might help.

$reqmthd = $ENV{'REQUEST_METHOD'};
$reqhost = $ENV{'HTTP_HOST'};
$requri = $ENV{'REQUEST_URI'};
$referer = $ENV{'HTTP_REFERER'};
$usragnt = $ENV{'HTTP_USER_AGENT'};
<snip>
print HTMLOG ("<br><b>$remaddr</b> banned $date $reqmthd $reqhost$requri \"$referer\" \"$usragnt\"\n");

Jim

Wizcrafts




msg:1508449
 2:54 am on Oct 3, 2003 (gmt 0)

$referer = $ENV{'HTTP_REFERER'};

I don't have any apostrophies in my code. Maybe that broke just the referer field. I'll retest it in my experimental directory.

Wizcrafts




msg:1508450
 3:49 am on Oct 3, 2003 (gmt 0)

Nope, I still cannot use that variable in my email codes. Everything else works fine, but including
$referer = $ENV{'HTTP_REFERER'}; in the script and calling it to for inclusion in the email, while allowing the email to be sent, still gives a 500 server error upon exiting. Here is my code for the email function, borrowed from one posted by another forumite a while ago:

$remreq = $ENV{REQUEST_URI};
$remaddr = $ENV{REMOTE_ADDR};
$usragnt = $ENV{HTTP_USER_AGENT};
$remmeth = $ENV{REQUEST_METHOD};
$remhost = $ENV{HTTP_HOST};
$referer = $ENV{'HTTP_REFERER'};
$date = scalar localtime(time);

open(MAIL, "¦/usr/sbin/sendmail -t") ¦¦ die "Content-type: text/text\n\nCan't open /usr/sbin/sendmail!";
print MAIL "To: xxx\@xxx\.xxx\n";
print MAIL "From: xxx\@xxx\.xxx\n";
print MAIL "Subject: You caught another one!\n";
print MAIL "The ip address: $remaddr was banned on $date \n";
print MAIL "The file requested was: $remreq\n";
print MAIL "The method used was: $remmeth\n";
print MAIL "The intruder's user agent was: $usragnt\n";
print MAIL "The remote host was: $remhost\n";
print MAIL "The referrer was: $referer\n";
# The above line's referer variable causes a 500 server error
close(MAIL);
exit;

The results are emailed to me with a blank referrer variable, and I get a 500 server error on screen.

Wiz

jdMorgan




msg:1508451
 3:58 am on Oct 5, 2003 (gmt 0)

Mail script discussion continues here [webmasterworld.com].

Wizcrafts




msg:1508452
 1:50 pm on Oct 6, 2003 (gmt 0)

The sendmail with the refer field problem has been solved here [webmasterworld.com], in message #6.

Wiz

balam




msg:1508453
 3:58 pm on Oct 6, 2003 (gmt 0)

Balam's FormMail Request RewriteRules (157 bytes)

jbMorgan's compressed rules (75 bytes)

jbMorgan's compressed rules with Wizcrafts modification (121 bytes)

I've been a lot of things in life, but I'm not sure if "compressed" has ever been one of them... :)

Wizcrafts




msg:1508454
 7:20 pm on Oct 10, 2003 (gmt 0)

I am having a problem with a particular RewriteRule. I want to send anybody using one particular UA to a special "banned" page and allow them access to nothing else. I have no problem already sending them to my 403 page. Ideally, this page would replace the custom 403 page they now get, but have the same effect as a 403. I want to make a point that they are banned, not just denied access.

Here is what I tried, that does not work:

Options +FollowSymLinks
RewriteEngine On

# This is the first Rewrite Cond and Rule

RewriteCond %{HTTP_USER_AGENT} ^User_Agent_to_ban$ [NC]
RewriteRule .* /includes/banned.html [L]

# other conditions follow, not covering this one
# Here is part of the main RewriteRule for other conditions:

RewriteRule !^(includes/403\.html¦/includes/banned\.html) - [F]

This rule hangs endlessly with Wannabrowser, unless I change the RewriteRule to:
RewriteRule !^(includes/403\.html) - [F,L] where it instantly goes to the custom 403. The banned.html does exist in the includes folder. I have also tried appending [mydomain...] to the rule, but it made no difference.

I have allowed /includes/banned\.html in my universal RewriteRule and in my allowit rule.

TIA, Wiz

closed




msg:1508455
 4:41 am on Oct 11, 2003 (gmt 0)

If you look carefully at the code for your RewriteRule, you should be able to see that there is a forward slash at the beginning for banned.html, but there isn't one for 403.html.

Wizcrafts




msg:1508456
 3:38 pm on Oct 11, 2003 (gmt 0)

If you look carefully at the code for your RewriteRule, you should be able to see that there is a forward slash at the beginning for banned.html, but there isn't one for 403.html.

BINGO!
Thanks Closed!

El Wizzo

closed




msg:1508457
 3:47 pm on Oct 11, 2003 (gmt 0)

You're welcome, El Wizzo. BTW, my asking price for solving your problem is 50% of your bingo winnings. ;)

Wizcrafts




msg:1508458
 4:23 pm on Oct 11, 2003 (gmt 0)

Does that include sharing my lottery ticket losses? I am on a roll: 100% wrong numbers!

Wizcrafts




msg:1508459
 4:35 pm on Oct 11, 2003 (gmt 0)

This is an great alternative page to my lengthly 403 page, which is served up to people or bots on my general unwanted list. This particular page is reserved for the guy who tries to spam guestbook scripts with a script name that we all write about on these forums. My detection technique is a secret passed down from generations of Romanian Gypsies (and fellow WebmasterWorld members).

I have tried browsing my website with this code in place, and not even the 403 page is available to it. As Judge Judy would say: Perfect!

claus




msg:1508460
 5:24 pm on Oct 11, 2003 (gmt 0)

I'm afraid i don't know said Judge, but it's a good point you mention; that it's possible to deny access in more than one way, depending on user-agent or another variable.

An alternative way is to make your 403 error document a script, or rather: Use a script as 403 error document. That way you can set up even more sophisticated rules, eg. using databases.

/claus

merrioc




msg:1508461
 1:26 am on Oct 12, 2003 (gmt 0)

This is my first post here so someone drop me a note if I break ettiquet.

Total(Gigs) 925.31

this is my total GB transfer so far this month (yes in 11 days)which is costing me a bloody fortune. I've posted my .htaccess that I just added today after reading through this thread (and the prior one) but since I don't understand mod_rewrite very well or regular expressions I hoping that some of the more SR members here can help me refine it.

what I am hoping to accoplish with my .htaccess
block downloaders/site rippers
block people that try to directly link to the content
people that have no reffer value
block EVERYONE that has the word "forum" in there reffer

what hasn't been accoplished that I would still like to
block reffers from specific TLD's (IE .jp .ch .nl)
exploit scan blocking (something simular to this thread but I can't find anything more specific or more explanatory [webmasterworld.com...]

RewriteEngine On
RewriteCond %{HTTP_REFERER} ^-?$ [NC]
RewriteCond %{HTTP_USER_AGENT} ^-?$ [NC]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(Auto¦Cop¦dup¦Fetch¦Filter¦Gather¦Go¦Leach¦Mine¦Mirror¦Pix¦QL¦RACE¦Sauger) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(site.?(eXtractor¦Quester)¦Snake¦ster¦Strip¦Suck¦vac¦walk¦Whacker¦ZIP) [NC,OR]
#RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦URL¦WebDAV¦Foundation).(Access¦Control¦MiniRedir¦Class) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^(BlackWidow¦Crescent¦Disco.?¦ExtractorPro¦HTML.?Works¦Franklin.?Locator¦HLoader¦http.?generic¦Industry.?Program¦IUPUI.?Research.?Bot¦Mac.?Finder¦NetZIP¦NICErsPRO¦NPBot¦PlantyNet_WebRobot¦Production.?Bot¦Program.?Shareware¦Teleport.?Pro¦TurnitinBot¦TE¦VoidEYE¦WebBandit¦WebCopier¦WEP.?Search¦Wget¦Zeus) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cherry.?picker¦e?mail.?(collector¦extractor¦magnet¦reaper¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Educate.?Search¦Full.?Web.?Bot¦Indy.?Library¦IUFW.?Web [NC,OR]
RewriteCond %{HTTP_USER_AGENT} httrack¦larbin¦NaverRobot¦Siphon¦SURF [NC,OR]
RewriteCond %{HTTP_USER_AGENT} efp@gmx\.net [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.?URL.?Control [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Miss.*g.*.?Locat.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.06\ \(Win95;\ I\) [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0\ \(compatible\ ;\ MSIE.? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0\ \(compatible;\ MSIE\ 5\.00;\ Windows\ 98$ [NC,OR]
# The next lines block NPBot by IP
RewriteCond %{REMOTE_ADDR} ^12\.148\.196\.(12[8-9]¦1[3-9][0-9]¦2[0-4][0-9]¦25[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^12\.148\.209\.(19[2-9]¦2[0-4][0-9]¦25[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^12\.175\.0\.(3[2-9]¦4[0-7])$ [OR]
RewriteCond %{REMOTE_ADDR} ^(203\.186\.145\.225¦218\.6\.10\.113¦68\.59\.94\.40¦66\.75\.128\.202)$ [OR]
RewriteCond %{REMOTE_ADDR} ^210\.192\.(9[6-9]¦1[0-1][0-9]¦12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^211\.(1[0-1][4-9])\. [OR]
RewriteCond %{REMOTE_ADDR} ^218\.([0-2][0-9]¦[3][0-1])\. [OR]
RewriteCond %{REMOTE_ADDR} ^218\.(5[6-9]¦[6-9][0-9])\. [OR]
# Start Cyveillance blocks
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.2(2[4-9]¦[3-4][0-9]¦5[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.118\.41\.(19[2-9]¦2[0-1][0-9]¦22[0-3])$ [OR]
# End Cyveillance blocks

RewriteCond %{HTTP_REFERER} q=guestbook [NC,OR]
RewriteCond %{HTTP_REFERER} iaea\.org [NC]
RewriteRule ^.*$ [badplace.com...]

# Forbid requests for exploits & annoyances
#
# Bad requests
RewriteCond %{REQUEST_METHOD}!^(GET¦HEAD¦POST) [NC,OR]
# CodeRed
RewriteCond %{REQUEST_URI} ^/default\.(ida¦idq) [NC,OR]
RewriteCond %{REQUEST_URI} ^/.*\.printer$ [NC,OR]
# Email
RewriteCond %{REQUEST_URI} (mail.?form¦form¦form.?mail¦mail¦mailto)\.(cgi¦exe¦pl)$ [NC,OR]
# MSOffice
RewriteCond %{REQUEST_URI} ^/(MSOffice¦_vti) [NC,OR]
# Nimda
RewriteCond %{REQUEST_URI} /(admin¦cmd¦httpodbc¦nsiislog¦root¦shell)\.(dll¦exe) [NC,OR]
# Various
RewriteCond %{REQUEST_URI} ^/(bin/¦cgi/¦cgi\-local/¦sumthin) [NC,OR]
RewriteCond %{THE_REQUEST} ^GET\ http [NC,OR]
RewriteCond %{REQUEST_URI} /sensepost\.exe [NC]
# Forbid if UA is a single word - case-insensitive, A-Z only
RewriteCond %{HTTP_USER_AGENT} ^[a-z]+$ [NC]
# Some exemptions though...
RewriteCond %{HTTP_USER_AGENT}!^ColdFusion$ [OR]
RewriteCond %{HTTP_USER_AGENT}!^DeepIndex$ [OR]
RewriteCond %{HTTP_USER_AGENT}!^FavOrg$ [OR]
RewriteCond %{HTTP_USER_AGENT}!^MantraAgent$ [OR]
RewriteCond %{HTTP_USER_AGENT}!^MARTINI$ [OR]
# Address harvesters
RewriteCond %{HTTP_USER_AGENT} ^(autoemailspider¦ExtractorPro) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^E?Mail.?(Collect¦Harvest¦Magnet¦Reaper¦Siphon¦Sweeper¦Wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (DTS.?Agent¦Email.?Extrac) [NC,OR]
RewriteCond %{HTTP_REFERER} iaea\.org [NC,OR]
# Download managers
RewriteCond %{HTTP_USER_AGENT} ^(Alligator¦DA.?[0-9]¦DC\-Sakura¦Download.?(Demon¦Express¦Master¦Wonder)¦FileHound) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Flash¦Leech)Get [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Fresh¦Lightning¦Mass¦Real¦Smart¦Speed¦Star).?Download(er)? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Gamespy¦Go!Zilla¦iGetter¦JetCar¦Net(Ants¦Pumper)¦SiteSnagger¦Teleport.?Pro¦WebReaper) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(My)?GetRight [NC,OR]
# Image-grabbers
RewriteCond %{HTTP_USER_AGENT} ^(AcoiRobot¦FlickBot¦webcollage) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Express¦Mister¦Web).?(Web¦Pix¦Image).?(Pictures¦Collector)? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image.?(fetch¦Stripper¦Sucker) [NC,OR]
# "Gray-hats"
RewriteCond %{HTTP_USER_AGENT} ^(Atomz¦BlackWidow¦BlogBot¦EasyDL¦Marketwave¦Sqworm¦SurveyBot¦Webclipping\.com) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (girafa\.com¦gossamer\-threads\.com¦grub\-client¦Netcraft¦Nutch) [NC,OR]
# Site-grabbers
RewriteCond %{HTTP_USER_AGENT} ^(eCatch¦(Get¦Super)Bot¦Kapere¦HTTrack¦JOC¦Offline¦UtilMind¦Xaldon) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(Auto¦Cop¦dup¦Fetch¦Filter¦Gather¦Go¦Leach¦Mine¦Mirror¦Pix¦QL¦RACE¦Sauger) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(site.?(eXtractor¦Quester)¦Snake¦ster¦Strip¦Suck¦vac¦walk¦Whacker¦ZIP) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} WebCapture [NC,OR]
# Tools
RewriteCond %{HTTP_USER_AGENT} ^(curl¦Dart.?Communications¦Enfish¦htdig¦Java¦larbin) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (FrontPage¦Indy.?Library¦RPT\-HTTPClient) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww¦lwp¦PHP¦Python¦www\.thatrobotsite\.com¦webbandit¦Wget¦Zeus) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦Internet¦URL¦WebDAV¦Foundation).(Access¦Explorer¦Control¦MiniRedir¦Class) [NC,OR]
# Unknown
RewriteCond %{HTTP_USER_AGENT} ^(Crawl_Application¦Lachesis¦Nutscrape) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^[CDEFPRS](Browse¦Eval¦Surf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Demo¦Full.?Web¦Lite¦Production¦Franklin¦Missauga¦Missigua).?(Bot¦Locat) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (efp@gmx\.net¦hhjhj@yahoo\.com¦lerly\.net¦mapfeatures\.net¦metacarta\.com) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Industry¦Internet¦IUFW¦Lincoln¦Missouri¦Program).?(Program¦Explore¦Web¦State¦College¦Shareware) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Mac¦Ram¦Educate¦WEP).?(Finder¦Search) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Moz+illa¦MSIE).?[0-9]?.?[0-9]?[0-9]?$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9][0-9]?.\(compatible[\)\ ] [NC,OR]
RewriteCond %{HTTP_USER_AGENT} NaverRobot [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*forum*.*\.com [NC]
RewriteRule ^.*$ [aaa.sitethatdoesstuffworsethenmine.com...]

closed




msg:1508462
 5:40 am on Oct 12, 2003 (gmt 0)

Welcome to WebmasterWorld [webmasterworld.com], merrioc!

I hoping that some of the more SR members here can help me refine it

This board says I'm a Junior Member, but since no one else has posted yet, I figured I might as well.

Okay, I guess I'll just go down the list:
block downloaders/site rippers

I'm not quite sure about who you want to block, so I'll ignore this for now.

block people that try to directly link to the content

Generally, you'll use something like this:

RewriteCond %{HTTP_REFERER} !(www\.)?mysite.com/ [NC]

That would check to see if the referrer is not from your site.

people that have no reffer value

Your code:

RewriteCond %{HTTP_REFERER} ^-?$ [NC]

is okay, except that you should remember to put an OR flag too when you have other RewriteConds because otherwise the AND flag is assumed.

block EVERYONE that has the word "forum" in there reffer

Your code:

RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*forum*.*\.com [NC]

is okay again, except for the fact that it will probably only block referrers that are .coms and whose domain names contain forum.

Since you want to block any referrer that contains forum, this should work for your purposes:

RewriteCond %{HTTP_REFERER} forum [NC]

block reffers from specific TLD's (IE .jp .ch .nl)

I'll let you start this one first. It's not that hard.

exploit scan blocking (something simular to this thread but I can't find anything more specific or more explanatory [webmasterworld.com...]

Not quite sure what you mean, so I'll ignore it for now. I'm not a preferred member, so I don't have access to the thread you quoted.

In case you're stuck, here are some references I'd recommend:
mod_rewrite [httpd.apache.org]
Regular expressions [etext.lib.virginia.edu]

Wizcrafts




msg:1508463
 12:51 pm on Oct 13, 2003 (gmt 0)
I have a followup question about message #112 in this thread.
I have this rule in .htaccess:
[code]
RewriteCond %{HTTP_USER_AGENT} ^User_Agent_to_be_banned$ [NC]
RewriteRule .* includes/banned.html [L]
[/code]
Yesterday the selected UA visited my website again, using POST as his method. Instead of getting a 200 and seeing my special message page, the log shows that he got a generic 405 page, "method not allowed." Should I add a method rule to the rewrite condition for GET and POST? I figured that my rule would work for any allowed method (GET,POST,HEAD,OPTIONS), but I guess that is not really the case.

Is this what I need to add:
[code]
RewriteCond %{REQUEST_METHOD} ^(GET¦POST)$
RewriteCond %{HTTP_USER_AGENT} ^User_Agent_to_be_banned$ [NC]
RewriteRule .* includes/banned.html [L]
[/code]

Wiz

This 122 message thread spans 5 pages: < < 122 ( 1 2 3 [4] 5 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved