homepage Welcome to WebmasterWorld Guest from 54.204.59.230
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

This 122 message thread spans 5 pages: < < 122 ( 1 2 [3] 4 5 > >     
A Close to perfect .htaccess ban list - Part 2
adriaant




msg:1508343
 11:46 pm on May 14, 2003 (gmt 0)

<modnote>
continued from [webmasterworld.com...]



UGH, bad typo in my original post. Here's the better version (I wasn't able to re-edit the older post?):

I'm trying to ban sites by domain name, since there are recently lots of reference spammers.

I have, for example, the rule:

RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*stuff.*\.com/.*$ [NC]
RewriteRule ^.*$ - [F,L]

which should ban any sites containing the word "stuff"
www.stuff.com
www.whatkindofstuff.com
www.some-other-stuff.com

and so on.

However, it is not working, so I am sure I did not setup a proper pattern match rule. Anyone care to advise?

[edited by: jatar_k at 5:06 am (utc) on May 20, 2003]

 

nancyb




msg:1508403
 7:36 pm on Sep 17, 2003 (gmt 0)

I have the following in htaccess

# Block libwww-perl except from AltaVista, Inktomi, and IA Archiver
RewriteCond %{HTTP_USER_AGENT} ^libwww-perl/[0-9] [NC]
RewriteCond %{REMOTE_ADDR}!^209\.73\.(1[6-8][0-9]¦19[01])\.
RewriteCond %{REMOTE_ADDR}!^209\.131\.(3[2-9]¦[45][0-9]¦6[0-3])\.
RewriteCond %{REMOTE_ADDR}!^209\.237\.23[2-5]\.
RewriteRule!^err403\.htm$ - [F]
# Block Java and Python URLlib except from Google
RewriteCond %{HTTP_USER_AGENT} ^(Python.urllib¦Java/?[1-9]\.[0-9]) [NC]
RewriteCond %{REMOTE_ADDR}!^216\.239\.(3[2-9]¦[45][0-9]¦6[0-3])\.

can anyone tell why the first hit gets a 200 and the second is 404? and what I need to do to correct it so both are 404?

65.49.178.17 - - [17/Sep/2003:10:40:52 -0400] "GET /xxx.htm HTTP/1.1" 200 14724 "-" "xxxxxxxxx_xxxxxxxx/0.1 libwww-perl/5.65"
65.49.178.17 - - [17/Sep/2003:10:34:02 -0400] "GET /xxxxxx/- HTTP/1.1" 404 7550 "-" "xxxxxxxxx_xxxxxxxx/0.1 libwww-perl/5.65"

thanks

closed




msg:1508404
 5:00 am on Sep 18, 2003 (gmt 0)

It looks to me like the code you posted has nothing to do with the log file entries, because the code checks to see if the UA begins with libwww... or Python.... or Java..., and there is no UA like that in your log file snippet.

I'd guess that there are no restrictions imposed on 65.49.178.17. The 404 was due to the fact that the document being retrieved was /xxxxxx/-, which is a malformed address. The 200 was due to the fact that /xxx.htm existed, and there were no access restrictions on 65.49.178.17.

I'm guessing you want to correct both log file entries so that a 403 status code (Forbidden) is returned. In that case, I'd change this line:

RewriteCond %{HTTP_USER_AGENT} ^libwww-perl/[0-9] [NC]

to this:

RewriteCond %{HTTP_USER_AGENT} libwww-perl/[0-9] [NC]

I removed the ^ from the first RewriteCond so that you check for the occurrence of libwww-perl/[0-9] anywhere in the UA, which is what happened here.

If you really do want to send back a 404, you'd have to modify the RewriteRule to use the R flag with a status code of 404. Since you want to block unwanted users, though, my guess was that you actually wanted to send back a 403.

nancyb




msg:1508405
 3:06 pm on Sep 18, 2003 (gmt 0)

thank you, closed!

I looked and looked at the code and it just didn't sink in and I completely missed the "-". Could be all the problems with my site/host over the last weeks and I'm brain dead from looking for problems - or - it could be I'm just as blind as a bat.

Either case, thank you so much for the clear explanation! :)

Wizcrafts




msg:1508406
 6:22 pm on Sep 18, 2003 (gmt 0)

Speaking from with a mental block, I post:

In my .htaccess file I have applied all of the suggestions found throughout this thread and everything is working fine.

One of the thorns in my side has been FormMail Phishers, so I have taken the now-famous Trap.pl, customized it and renamed it "formmail.pl," allowed that file in my RewriteRule and it works fine to ban Phishers. Most Phishers come at me about 8 to 10 times in a row, with various spellings, extensions, and directory names, but have always been caught in my trap when they type in "formmail.pl"

However, while reading yesterday's web log I found a FormMail phisher that evaded my trap by only looking for variations of this exact spelling: cgi-bin/FormMail.pl (and .cgi). My ban-bad-bots trap is named formmail.pl and was not triggered because it is all lowercase, but he did get 403's by my RewrightCond for form.?mail [nc,or].

I tried adding this line to my .htaccess but it does not redirect the request to formmail.pl:
RedirectMatch permanant cgi-bin/FormMail\.pl cgi-bin/formmail.pl.
I also commented out the other conditions that would have caught this request and 403'd it. Every Wannabrowser attempt was met with a 403, but no redirects. Wannabrowser is not blocked in my .htaccess.

Can anybody help me straighten out the error so I can forward requests for "FormMail.pl" to "formmail.pl"? If I figure it out first I will post the working code-line later.

TIA, Wiz

closed




msg:1508407
 7:25 pm on Sep 18, 2003 (gmt 0)

nancyb: You're welcome so much for the clear explanation! :P Luckily, you edited the lines from the log file correctly so that the file and directory names remained anonymous, keeping the malformed address in place.

Wizcrafts: You should replace permanant with permanent. You could also use 301 instead.

Wizcrafts




msg:1508408
 7:36 pm on Sep 18, 2003 (gmt 0)

Closed;
I rectified the spelling error but I still get a 403 when I try to GET FormMail.pl. Any other ideas?

Here is the applicable RewriteCond and RewriteRule affecting FormMail:

RedirectMatch 301 cgi-bin/FormMail\.pl cgi-bin/formmail.pl

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{REQUEST_URI} formmail\.(cgi¦php)$ [NC]

RewriteRule!^(includes/403\.html¦cgi-bin/MKCounter\.cgi¦robots\.txt¦contact-info\.html¦kissthis\.html¦cgi-bin/contact-info\.cgi¦cgi-bin/contact-list\.pl¦cgi-bin/banbadbots\.cgi¦cgi-bin/formmail\.pl¦cgi-bin/FormMail\.pl¦bait/honeypot\.html¦bait/\w*\.html¦bait/contact-info\.cgi) - [F]

I figured it out myself!

When a request comes for a file in my cgi-bin and I tried to redirect that to cgi-bin/formmail.pl, I was actually telling the searcher to look in cgi-bin/cgi-bin/ for formmail.pl. I got it to work by dropping the cgi-bin/ in the destination file!

Wiz

[edited by: Wizcrafts at 7:59 pm (utc) on Sep. 18, 2003]

closed




msg:1508409
 7:59 pm on Sep 18, 2003 (gmt 0)

So it actually works?

From [httpd.apache.org ]:

RedirectMatch
Syntax: RedirectMatch [status] regex URL

I was going to say that you should change your third input to something that starts with http://.

Added: Never mind. I got confused between URL and URI.

[edited by: closed at 8:08 pm (utc) on Sep. 18, 2003]

claus




msg:1508410
 8:03 pm on Sep 18, 2003 (gmt 0)

I'm just wondering why you don't do this in stead:

Options +FollowSymLinks
RewriteEngine On
RewriteCond %{REQUEST_URI} (.?mail.?form¦form¦(GM)?form.?.?mail¦.?mail)(2¦to)?\.?(asp¦cgi¦exe¦php¦pl¦pm)?$ [NC,OR]
RewriteRule .* /path-to/bad-bot-script.pl [L]

If your bad-bot script bans them anyway, it seems odd to have an additional level of banning. Then it would seem more efficient to just dump them directly in the trap, thus getting them banned instantly.

My condition for formmail catches a few more than the one you posted, it's documented by balam here (msg #6): [webmasterworld.com...]

/claus


BTW: this link is really valuable, it's Engelschalls guide to url rewriting, it's better than the official Apache docs imho: [engelschall.com...]

[edited by: claus at 8:05 pm (utc) on Sep. 18, 2003]

Wizcrafts




msg:1508411
 8:03 pm on Sep 18, 2003 (gmt 0)

Here is how I got it to redirect correctly:

RedirectMatch cgi-bin/FormMail.pl formmail.pl

No 301 needed, it generates a 302 found response as the trespasser is viewing my trap text and getting banned.

This stuff can drive you nuts trying to get exact paths and syntax. I read the Apache docs and still had to figure it out by trial and error (heavy on the error side)

Thanks Claus, I'll try to implement that
Wiz

Wizcrafts




msg:1508412
 9:04 pm on Sep 18, 2003 (gmt 0)

Well, I have implemented the ban-all-phishers rule that Balam and Claus posted and it works.

I have found that there are times when the same IP address gets added to my ban list multiple times and I anticipate that this is going to happen more, since the FormMail phishers hit you 6 to 10 times at once, looking for different file names and paths, all of which are now banned by the all-inclusive rule.

Does anybody know how to edit the trap script to only add an IP address once, forever, no matter how many times they land on the ban script?

Here is the banning script section in question:

# trap.pl: upload in ASCII mode and CHMOD 755.

# This is the only variable that needs to be modified. Replace it with the absolute path to your root directory.
$rootdir = "$ENV{DOCUMENT_ROOT}";

# Grab the IP of the bad bot
$visitor_ip = $ENV{'REMOTE_ADDR'};
$visitor_ip =~ s/\./\\\./gi;

# Open .htaccess file
open(HTACCESS,"".$rootdir."/\.htaccess") ¦¦ die $!;
@htaccess = <HTACCESS>;
close(HTACCESS);

# Write banned IP to .htaccess file
open(HTACCESS,">".$rootdir."/\.htaccess") ¦¦ die $!;
print HTACCESS "SetEnvIf Remote_Addr \^".$visitor_ip."\$ ban\n";
foreach $deny_ip (@htaccess) {
print HTACCESS $deny_ip;
}
close(HTACCESS);

I think the multiple listings occur because I have allowed access to formmail.pl and my trap script in my master rewrite rule. Maybe I can now remove that allowance since the formmail rule is totally separate. I'll see.

kevinpate




msg:1508413
 5:10 pm on Sep 21, 2003 (gmt 0)

RewriteCond %{REMOTE_ADDR}!^209\.73\.(1[6-8][0-9]¦19[01])\.
RewriteCond %{REMOTE_ADDR}!^209\.131\.(3[2-9]¦[45][0-9]¦6[0-3])\.
RewriteCond %{REMOTE_ADDR}!^209\.237\.23[2-5]\.

I'm not sure I'm actually getting my brain fully wrapped around the banning of IP ranges. Actually, based on some recent 500 errors, I'm down right positive I don't have a good grip on it when I try it with some other ranges.

In the above example, do the lines block these ranges:

209.73. 160 through 191
209.131. 32 through 63
209.237. 232 through 235

closed




msg:1508414
 5:53 pm on Sep 21, 2003 (gmt 0)

No. The lines you just mentioned would only check to see whether the IP address is not in those ranges. The RewriteRule that comes after it would do the blocking, and would be applied only if the preceding RewriteConds were met.

If you wanted to block addresses in those ranges, you would remove the exclamation point. Here's a reference on regular expressions:
[etext.lib.virginia.edu ]

FYI, in the code you quoted above, you'd have to put a space before each exclamation point, and replace the vertical pipes with the ones on your keyboard because this forum's software modifies the formatting of the code.

Constantin




msg:1508415
 3:50 pm on Sep 26, 2003 (gmt 0)

Thanks to webmasterworld, I now have a rather spiffy .htaccess ban file. Lots of UAs are banned, yet more and more crop up. Worse, UAs are starting to get spoofed effectively... It won't be long before telling a bot from a user will become very difficult.

At some point I wonder if the .htaccess file is going to get bogged down with all the banned UAs, IPs, etc. Has anyone measured the impact on server performance?

Secondly, I wonder if using a bot trap isn't more efficient. Yes, there are some risks, like banning the wrong folks, but that risk can be minimized by hiding the trigger. I would think that a simple deny from IP address would be much more effective than a long list of UA's?

Then recycle the .htaccess file with a cron job every couple of days to keep it lean and mean?

Nick_W




msg:1508416
 3:53 pm on Sep 26, 2003 (gmt 0)

Can someone post this .htaccess file in all it's glory?

I really want to use it and it would be soooo helpful to have a summary for the likes of me ;)

Nick

jdMorgan




msg:1508417
 4:17 pm on Sep 26, 2003 (gmt 0)

Wizcrafts,

> Does anybody know how to edit the trap script to only add an IP address once, forever, no matter how many times they land on the ban script?

This should not be necessary. If you get a request from an IP address and it triggers the script, the script adds that IP address to your .htaccess file ban list. If that IP address comes back, it should then get either the standard 403-Forbidden server page, or your custom 403 page if you've defined one. Therefore, it should not be able to re-invoke the script.

The only exception to this is the case where a bad guy has multiple "sessions" open, hitting your server a second or third time before the bot script has had a chance to finish running (banning the IP address) for the first request. In this case, you may see a few duplicate entries in your .htaccess ban list.

It is possible to filter these out, but the script would have to be modified to search through previous ban list entries and compare the IP addresses to the current one. This slows down the script, and therefore you would get even more duplicate entries because the time window for duplicates would be increased by the longer filtering script processing time. So, the filtering technique very soon becomes self-defeating, and I recommend against it.

On the other hand, you may have a problem with your set-up, so let me qualify the above discussion a little more fully: If an IP addresses is still listed in your .htaccess ban list and if it hits the trap and is written again to your ban list more than two or three seconds after the initial entry, then there is something wrong with your installation. Once an IP is banned (which might take two seconds at most), then it should never appear again in the ban list after that two seconds. Instead, you should see that it always gets 403-Forbidden responses after it has been added to the ban list in your .htaccess file. If this is not the case, then something isn't set up correctly.

Hopefully, this explanation is clear enough to be helpful...

Jim

jdMorgan




msg:1508418
 4:34 pm on Sep 26, 2003 (gmt 0)

Nick_W,

Everybody's .htaccess is likely to be different, and to need to be different. Our sites are all different, and therefore they attract different attackers.

For info on the script that's being mentioned here, start at this recent thread and also read the previous ones: [webmasterworld.com...]

Constantin,

Sometimes it's more efficient to ban by user-agent if that user-agent is being widely-used at the time. When a spoofed user-agent is used, then you must ban by IP address, but that ban list can quickly grow too large. So "pruning" the list, either manually or with a cron job is a good idea. However, some sites may need to prune daily, and others can go for weeks or even months without cleaning up the list, depending on how much bad-bot traffic they are getting.

Personally, I've noticed that attacks slowed down a LOT after I'd had the script in place for several months. The harvesters like to sneak in and grab stuff without being noticed. A 403 response tells them that they've been noticed, and that it's quite possible their IP addresses and proxy addresses are being reported. So, the smart ones don't come back...

Everybody has to figure out the most efficient way to use user-agent blocks, IP address blocks, and bad-bot blocking scripts on their own sites. I doubt there's a good one-size-fits-all recommendation for all sites.

Jim

Wizcrafts




msg:1508419
 5:39 pm on Sep 26, 2003 (gmt 0)

Jim;

You are correct (as usual)! I was erroneously allowing rule-banned spiders to access the self-banning script, among other files that poison, 403, record, entrap them, etc. I have since taken the trap script out of that allowed list and placed it under its own rewrite conditions that only deal with FormMail Phishers.
-------------------------------------------------------------

Nick asked to see a complete .htaccess that exemplifies what is being discussed here. I will post mine, with a few items renamedd out to protect the innocent (my domain) and items that are specific to my installation in italics. Also read the note below the codes.

<Files *>
order deny,allow
deny from env=ban
allow from all
</Files>

<Files .htaccess>
deny from all
</Files>

XBitHack On
ErrorDocument 403 /[i]includes/403.html[/i] # my filename and path
DirectoryIndex index.html index.htm index.shtml index.php /[i]noread.html[/i] #my file

<Files *>
<LimitExcept GET POST>
deny from all
</LimitExcept>
</Files>

order deny,allow
deny from 12.219.232.74
deny from 24.53.200.12
deny from 24.188.211.3
deny from 61.4.64.0/20
deny from 62.253.166.153
deny from 65.33.10.192
deny from 65.57.163.78
deny from 66.36.240.135
deny from 66.36.246.127
deny from 66.72.195.144
deny from 66.76.144.219
deny from 66.119.34.39
deny from 66.250.125.195
deny from 68.42.21.162
deny from 142.177.144.148
deny from 170.224.224.38
deny from 200.176.32.214
deny from 203.194.146.175
deny from 204.234.17.35
deny from 206.135.194.194
deny from 207.134.171.4
deny from 210.192.120.74
deny from 210.192.96.0/17
deny from 212.138.47.18
deny from 213.221.116.114
deny from 216.93.191.2
deny from 217.21.117.121
deny from 217.78.
deny from 220.73.25.68
deny from 220.73.165.
deny from 220.99.112.2
allow from all

Options +FollowSymLinks
RewriteEngine On

RewriteCond %{REQUEST_URI} (.?mail.?form¦form¦(GM)?form.?.?mail¦.?mail)(2¦to)?\.?(asp¦cgi¦exe¦php¦pl¦pm)?$ [NC]
RewriteRule .* MyDomain/cgi-bin/MyTrapFileName [L] # names have been changed

# The following are universal rules anybody can use in .htaccess:

RewriteCond %{HTTP_USER_AGENT} ^(BlackWidow¦Crescent¦Disco.?¦ExtractorPro¦HTML.?Works¦Franklin.?Locator¦HLoader¦http.?generic¦Industry.?Program¦IUPUI.?Research.?Bot¦Mac.?Finder¦NetZIP¦NICErsPRO¦NPBot¦PlantyNet_WebRobot¦Production.?Bot¦Program.?Shareware¦Teleport.?Pro¦TurnitinBot¦TE¦VoidEYE¦WebBandit¦WebCopier¦WEP.?Search¦Wget¦Zeus) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cherry.?picker¦e?mail.?(collector¦extractor¦magnet¦reaper¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Educate.?Search¦Full.?Web.?Bot¦Indy.?Library¦IUFW.?Web [NC,OR]
RewriteCond %{HTTP_USER_AGENT} httrack¦larbin¦NaverRobot¦Siphon¦SURF [NC,OR]
RewriteCond %{HTTP_USER_AGENT} efp@gmx\.net [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.?URL.?Control [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Miss.*g.*.?Locat.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.06\ \(Win95;\ I\) [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0\ \(compatible\ ;\ MSIE.? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0\ \(compatible;\ MSIE\ 5\.00;\ Windows\ 98$ [NC,OR]
# The next lines block NPBot by IP
RewriteCond %{REMOTE_ADDR} ^12\.148\.196\.(12[8-9]¦1[3-9][0-9]¦2[0-4][0-9]¦25[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^12\.148\.209\.(19[2-9]¦2[0-4][0-9]¦25[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^12\.175\.0\.(3[2-9]¦4[0-7])$ [OR]
RewriteCond %{REMOTE_ADDR} ^(203\.186\.145\.225¦218\.6\.10\.113¦68\.59\.94\.40¦66\.75\.128\.202)$ [OR]
RewriteCond %{REMOTE_ADDR} ^210\.192\.(9[6-9]¦1[0-1][0-9]¦12[0-7])\. [OR]
RewriteCond %{REMOTE_ADDR} ^211\.(1[0-1][4-9])\. [OR]
RewriteCond %{REMOTE_ADDR} ^218\.([0-2][0-9]¦[3][0-1])\. [OR]
RewriteCond %{REMOTE_ADDR} ^218\.(5[6-9]¦[6-9][0-9])\. [OR]
# Start Cyveillance blocks
RewriteCond %{REMOTE_ADDR} ^63\.148\.99\.2(2[4-9]¦[3-4][0-9]¦5[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.118\.41\.(19[2-9]¦2[0-1][0-9]¦22[0-3])$ [OR]
# End Cyveillance blocks

RewriteCond %{HTTP_REFERER} q=guestbook [NC,OR]
RewriteCond %{HTTP_REFERER} iaea\.org [NC]

RewriteRule !^(includes/403\.html¦cgi-bin/various_filenames\.pl¦various_filenames\.html) - [F]
# alternate RewriteRule without allowing access to custom 403 or trap pages, or cgi scripts:
# RewriteRule .* - [F]

Please note that this Board changes the vertical pipe ¦ from a solid pipe to a broken-line pipe. You will need to retype the vertical pipe on your keyboard to replace any broken pipes displayed here, if you choose to use these rules. Also, I see that some of the Rewrite lines have been word-wrapped. They were typed on one line, ending with the [OR] or [NC,OR] switches. Also, there should be a space in my first RewriteRule, between RewriteRule and the!^

Wiz

jdMorgan




msg:1508420
 8:26 pm on Sep 26, 2003 (gmt 0)

Wiz,

Have you verified that your fixed list of IP address is in fact blocked?

Another member on this board has reported problems when there are two "Order" sections - with the contents of the second one being ignored.
If you also see this problem, simply combine the two "deny from" lists under one "Order" directive. For example:

<Files .htaccess>
deny from all
</Files>

<Files *>
<LimitExcept GET POST>
deny from all
</LimitExcept>
</Files>

# Allow everybody to access custom 403 page and robots.txt
SetEnvIf Request_URI "^(/custom403\.html¦/robots\.txt)$" allowit
# Deny based on environment variables set above and IP addresses.
<Files *>
Order Deny,Allow
Allow from env=allowit
Deny from env=ban
Deny from 12.219.232.74
Deny from 24.53.200.12
<snip>
deny from 220.73.165.
deny from 220.99.112.2
</Files>

I'm not sure if this is a widespread problem or not, but maybe the above will help.

Jim

Wizcrafts




msg:1508421
 8:56 pm on Sep 26, 2003 (gmt 0)

Jim, it's funny you should ask that, because after I created this rule:
RewriteCond %{REMOTE_ADDR} ^210\.192\.(9[6-9]¦1[0-1][0-9]¦12[0-7])\. [OR], I noticed that a creepy crawler from 210.192.120.74 snuck in despite it, so I created a simple deny from rule for both that IP and the Host IP range, ala:
deny from 210.192.120.74
deny from 210.192.96.0/17
That definitely killed off all further contacts from anything in that range. Do you see anything wrong with my regular expressions in the RewriteRule for that IP range (210.192.96.0 - 210.192.127.255) that would cause it to allow 210.192,120.74? If so, I'd like to know.

I have seen 403s for IPs in the second and third block lists, but I like the idea of grouping common items in the same catagories. I'll get to that asap.

Thanks for demonstrating the allow from ruleset below:
# Allow everybody to access custom 403 page and robots.txt
SetEnvIf Request_URI "^(/custom403\.html¦/robots\.txt)$" allowit

Wiz

jdMorgan




msg:1508422
 9:13 pm on Sep 26, 2003 (gmt 0)

Wiz,

> a creepy crawler from 210.192.120.74 snuck in despite it

I don't see anything wrong with that RewriteCond or the regex in it, but look at the whole RewriteCond list and make sure you don't have a missing [OR], an unwelcome [OR] on the last RewriteCond, or some other kind of syntax or "structural" error.

Jim

Wizcrafts




msg:1508423
 3:10 pm on Sep 29, 2003 (gmt 0)

Q: Why would a POST method to my FormMail spam trap not spring the trap?

My logs shows POSTs to my IP trap script, but the IP did not get self-banned; I had to do it manually. Can anybody show me a Rewrite condition to add to, before, or after this RewriteCond to include POSTs? Right now only GETs trigger the script. However, POSTs to FormMail.anything do get 403s, just not auto-banned.

RewriteCond %{REQUEST_URI} (.?mail.?form¦form¦(GM)?form.?.?mail¦.?mail)(2¦to)?\.?(asp¦cgi¦exe¦php¦pl¦pm)?$ [NC]
RewriteRule .* path_to_my_trap_script [L]

BTW: I'd like to see an example of a mod to trap.pl that sends an email when the trap is sprung, if anybody has written such an addition.

Thanx, Wiz

jdMorgan




msg:1508424
 3:38 pm on Sep 29, 2003 (gmt 0)

Wiz,

You've got something else interfering with your call to the trap -- maybe a preceding <limit> statement that is rejecting POSTs before they even get to the trap invocation.

stapel posted the modification you're looking for last year. Try here [webmasterworld.com].

Jim

Wizcrafts




msg:1508425
 4:04 pm on Sep 29, 2003 (gmt 0)

Jim;
I have this just ahead of the Rewrite section, which I think I got from you:

<Files *>
<LimitExcept GET POST>
deny from all
</LimitExcept>
</Files>

I also have this condition several lines down from the formmail section, which leads to my main [F] rules:

RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.06\ \(Win95;\ I\) [OR]

Could this be the cause of POST not recording an IP on the trap, since this is the UA?
Oddly, I have second trap named formmail.pl that gives a 200 found response to POST, then does nothing but present its message, else unless they use GET.

jdMorgan




msg:1508426
 4:10 pm on Sep 29, 2003 (gmt 0)

Wiz,

Your LimitExcept won't stop POST, because it allows GET and POST and I think you're saying the user-agent block is after the call to your script, so neither of those sound like the problem.

Some hosting services intercept formmail queries before they even get to the hosted account's level. This is often the case when the host does not allow customers to use formmail.

Jim

Wizcrafts




msg:1508427
 4:26 pm on Sep 29, 2003 (gmt 0)

Jim;
I am able to use POST to my actual mailer script, and FormMail is allowed, but must be approved patched versions. I actually helped my host by posting information about FormMail exploits and what to do about preventing abuse. I will ask my host if he is blocking incoming POSTs. I never thought of that before.

In the meantime, here is the log of the attempt that was logged but not banned:

152.163.252.70 - - [29/Sep/2003:03:02:07 -0400] "POST /cgi-bin/FormMail.pl HTTP/1.0" 302 191 "-" "Mozilla/4.06 (Win95; I)"
152.163.252.100 - - [29/Sep/2003:03:02:07 -0400] "POST /cgi-bin/FormMail.cgi HTTP/1.0" 403 4105 "-" "Mozilla/4.06 (Win95; I)"

This was reported to abuse at aol.com.
Note that I have forwarded calls to "FormMail.pl" to be accepted by another script, which is supposed to record the IP for inclusion in the ban env. That first POST hit the trap but didn't get recorded.

claus




msg:1508428
 4:40 pm on Sep 29, 2003 (gmt 0)

>> didn't get recorded

Mozilla/4.06 (Win95; I)

That's not nice...perhaps we should look into escaping some chars in the bot-trap.

/claus

Wizcrafts




msg:1508429
 5:07 pm on Sep 29, 2003 (gmt 0)

Claus;
What am I missing here? The trap is Key_Master's trap.pl, which only cares about IP addresses, not User Agents. I have that UA banned in my list of conditions as follows:

RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.06\ \(Win95;\ I\) [OR]

Furthermore, the attempt coming from the same UA to POST to FormMail.cgi (non-existant) got my custom 403 page, which is definitely covered by the above rule. The problem is that the RewriteCond well ahead of this rule is supposed to divert calls to any variation of FormMail to the trap script, not the 403 page. After the IP gets banned, then they should get my 403 if they come back.

All of my pertinent codes are posted on the thread.

Wiz

jdMorgan




msg:1508430
 5:47 pm on Sep 29, 2003 (gmt 0)

Wiz,

Well, I'm stumped, but it's very interesting that the first POST attempt in your log entry shown above gets a 302-Moved Temporarily redirect. How/why is that happening? The answer might provide a clue to your larger problem.

Typically, as you show in your rewrite code snippet, an internal (transparent) rewrite is used to pass the request to the script, and not an external 301 or 302 redirect. So, that log entry is curious.

Jim

Wizcrafts




msg:1508431
 6:42 pm on Sep 29, 2003 (gmt 0)

Jim;

That redirect is caused by this line:

RedirectMatch cgi-bin/FormMail.pl formmail.pl

I added it because A) the all-inclusive IP ban line is not working, and B) I see many more requests to FormMail.pl, than to formmail.pl. That's why I created a 302 redirect to the lowercase filename, which is also a trap script. The log I showed shows that the redirect did work, but the IP was neither recorded nor banned.

Wiz

claus




msg:1508432
 6:45 pm on Sep 29, 2003 (gmt 0)

>> trap is Key_Master's trap.pl, which only cares about IP addresses

My bad, sorry. The ";" is not the problem then.

/claus

Wizcrafts




msg:1508433
 7:35 pm on Sep 29, 2003 (gmt 0)

I'm running some wannabrowser tests right now and this is what I have just discovered:

First, I have two of filenames in question in my cgi-bin.
When I use Wannabrowser to GET FormMail.cgi, or php, or Form-Mail.cgi, I get my custom 403 page, but not banned. When I type the search for FormMail.pl, or formmail.pl, both get banned, because they are both traps and both file names exist on my server.

Second, and best of all, when I type in a search for a non-existent filename in the cgi-bin I also get a 403! I used cgi-bin/nonexistent.php as the filename. There is nothing in my .htaccess that I know of that should cause a 403 instead of a 404. I am going to go over the htaccess file word by word to try to find out why this is happening.

More to come...

More: I just checked the permission I had set on my cgi-bin and found them to be 751. I chmodded to 755 and I now get a 404 error for a file not found, instead of 403 forbidden.

Final test: formmail.php now gets banned as designed! The whole problem was a lack of read permission for the World group. I believe I set it at 751 to prevent outsiders from reading the scripts and stealing email addresses, etc (months ago). I have since learned to secure the individual files with 711 permissions, which work just fine.

Thanks to all who tried to help in this unusual situation.

Wiz

This 122 message thread spans 5 pages: < < 122 ( 1 2 [3] 4 5 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved