Gorufu, littleman, Air, SugarKane? You guys see any errors or better ways to do this....anybody got a bot to add....before I stick this in every site I manage.
Feel free to use this on your own site and start blocking bots too.
(the top part is left out)<Files .htaccess>
deny from all
</Files>
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.]\.your-site.com.* - [F]
Who needs books !
Excellent data...
I wonder what the result would be performance-wise with four RewriteCond lines: One each for start-anchored, end-anchored, fully-anchored, and unanchored pattern strings. I use this method to keep the patterns organized by type, and to keep them neat and easy to maintain.
Thanks for the test - Very useful.
Jim
If I'm reading your code correctly, then you're demonstrating two things:
a) The response time difference between the two .htaccess files makes roughly 10%.
b) Each call takes between 0.002 and 0.006 seconds, depending on the load of the machine.
Combining those two, we're talking about an average additional overhead caused by multiple RewriteConds of 0.0004 seconds (1/2500 seconds) per request, on a somewhat aged machine.
Since you're always fetching the same file from the disk cache, the remaining server side overhead is very low (in contrast to a real life situation, where each request may cause a different set of files to be loaded from disk first), so we can assume that the difference is indeed caused by the different rule sets. The comments in your code talk about parsing the HTML, but I don't understand enough Perl to see if that really happens.
In summary, I'd prefer the maintenance friendly multiple-rule version any time, if it only costs me such a small price in terms of response time. Of course, I'm not serving millions of requests per day, so your mileage may vary.
comments in your code talk about parsing the HTML, but I don't understand enough Perl to see if that really happens.
No, no parsing going on since its irrelevant for the benchmarking.
on a somewhat aged machine
Donīt insult my trusted old linux box. I cannot guarantie for any DoS attacks it will launch on its own ;)
Since you're always fetching the same file from the disk cache, the remaining server side overhead is very low [...], so we can assume that the difference is indeed caused by the different rule sets
That was the idea behind the admittedly artificial setup.
I'd prefer the maintenance friendly multiple-rule version any time
As is quite often the case, itīs a tradeoff between speed and maintainability.
I followed this thread for quite some time since our server gets harassed by those "evil bots" as well. However, I couldn't quite decide to take action until the posting of jdMorgan which was almost a how-to.
However, having done this I run into the first trouble because my Apache 1.3 says:
Options FollowSymLinks or SymLinksIfOwnerMatch is off which implies that RewriteRule directive is forbidden
Simply adding a "FollowSymLinks on" on top of the htaccess doesn't work.
Any advice?
On a sidenote: I found some packages like "Sugerplum" and "robotcop" which promise to automize some of the functions intended by this htaccess. Any expereinces with these?
Options [httpd.apache.org] +FollowSymLinks
For that to works you need to have at least
AllowOverride [httpd.apache.org] Optionsprivileges. Those are set in the server config, virtual host context.
when spam comes in, i check the actual company site to get their genuine email addresses then manually add their email addresses to a mysql database. this means only genuine mailboxes get listed on the page and not the yahoo or hotmail addresses the spam is often sent from.
i've also added a simple browser check to the php page so that if an IE / Netscape / Opera user visits the page they will only see a normal forbidden message .... well, that's the theory, but i've not been able to test it yet. just need a "browser" or something that will let me set the UA to whatever i want .....
Thanks for the browscap.ini & sample IIS code, which I got from your site! I want to make sure I understand their use. I implement the browscap.ini (or whatever parts of it I want), and then I implement the code in global.asa for each robot I want to ban? Or just the one block of code gets revised to include each robot to be banned? (Or -- is every robot in the browscap.ini banned, so I should only include those which I wish to ban?) My partner is much more of a web programmer than I am and would take care of this, but I want to make sure I understand what needs to be done first!
Thanks a lot,
Snark
ErrorDocument 404 /404.htm
ErrorDocument 400 /404.htm
ErrorDocument 403 /404.htm
ErrorDocument 501 /404.htm
ErrorDocument 502 /404.htm
ErrorDocument 503 /404.htm
<FilesMatch "htm([l])*$">
ForceType application/x-httpd-php
</FilesMatch>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetSeer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Indy [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSFrontPage [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ping [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^webcollage [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule ^.* - [F,L]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.]\.your-site.com.* - [F]
Thanks
Anni
First post!
Since I run a virtual server with a number of different domains, it seems to me it would make more sense to put my list of forbidden UAs in the httpd.conf file, rather than try to replicate them in .htaccess on each domain's document root. Are there any caveats or special directions I should follow before I proceed?
Thanks!
Randy
[edited by: jatar_k at 12:04 am (utc) on Sep. 16, 2002]
[edit reason] no sigs please [/edit]
As shown in post #77 [webmasterworld.com] putting your RewriteRules in httpd.conf is indeed faster and the way to go when you have access to it.
However, this will not solve the problem of applying those rules to all virtual servers. You cannot just put the rewriting code in the main section and expect it to work for all virtual servers. For an explanation on this see API Phases [httpd.apache.org] in the mod_rewrite URL Rewriting Engine documentation.
So, after [...] Apache has determined the corresponding server (or virtual server) the rewriting engine starts processing of all mod_rewrite directives from the per-server configuration in the URL-to-filename phase.
my emphasis
Thereīs also a thread How (and Where) best to control access [webmasterworld.com] that you might want to read on this topic. If you have mod_perl you might want to use the solution mentioned in this thread. Ask carfac [webmasterworld.com] for the modified version of BlockAgent.
And as a sidenote. Do not drop any URLs. Do not use a signature.
# -FrontPage-
IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*
DirectoryIndex index.html index.htm index.php index.phtml index.php3
# AddType application/x-httpd-php .phtml
# AddType application/x-httpd-php .php3
# AddType application/x-httpd-php .php
#
# Action application/x-httpd-php "/php/php.exe"
# Action application/x-httpd-php-source "/php/php.exe"
# AddType application/x-httpd-php-source .phps
<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName www.XXXXXX.com
AuthUserFile /www/XXXXXX/_vti_pvt/service.pwd
AuthGroupFile /www/XXXXXX/_vti_pvt/service.grp
Should I remove this before pasting bans or simply add?
Thank you
Can I use this Rewrite stuff to block FrontPage from downloading my site? (I know the educators can still get my stuff from their browser's cache, etc, etc, but it would be nice to make them work at stealing, rather than having it be so easy, ya know?)
Thanks!
[edited by: jatar_k at 4:44 pm (utc) on Mar. 13, 2003]
Dunno if that helps.
The trap.pl script blocks thier ip address from further access to your website. This is very safe since "_vti_inf.hmtl" is only requested by Frontpage.
Works great!
"Edge" said:
When Frontpage first accesses a web site, the file _vti_inf.hmtl is requested.
I'm still waiting to hear from my host (this being the weekend) about whether "mod_rewrite" is available to me, but, in the meantime, I know that "Redirect" works. So could I do a Redirect, something like:
Redirect /_vti_inf.hmtl [purplemath.com...]
...to get rid of the FrontPage bums?
-----ten minutes later-----
I just tried the above line in my .htaccess file, and FrontPage was still able to download whatever it wanted from Purplemath into one of my other "webs". *sigh*
So tell me more about this script thingy...?
Sorry about that.
But I just tried again, using the proper spelling, and it still didn't work.
Oh, well. About that script you mentioned...?
First, install Apache::BlockAgents for each VH, and have them all point to the same bad_agent.txt- thus you only have one file to update for all hosts. (Note that all copies of this I have found on the web have perl errors in them- you will have to tweak that code to make it work at all.)
Then, make a copy of BlockAgents, modify the code a bit to handle IP's instead of agents, rename it BlockIP (or something!)and make a master bad_ip.txt file.
Third, get that trap.pl script, and modify that to write to bad_ip.tx rather than .htaccess. I further modified trap to it day/time stamps each entry, so I can clean it out every week.
This method is REALLY fast, and painless once set up (although set-up is a B***H!) It will work across all your VH, and if someone gets to onbe VH, they get locked out of all of them!
dave
won't work always. had this one today, grabbed some hundred pages from my beloved site:
p5084d1b1.dip.t-dialin.net - - [25/Sep/2002:13:34:40 +0200] "GET /_omitted.htm HTTP/1.0" 200 2373 www.mydomain.net "_omitted.htm" "Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)" "-"
So, this might be better as far as I can see:
RewriteCond %{HTTP_USER_AGENT} .*httrack.* [NC,OR]
Besides, HTTrack seems to respect robots.txt
I'd really like to block these fiends as well :)
thx
tom
So, this might be better as far as I can see:
RewriteCond %{HTTP_USER_AGENT} .*httrack.* [NC,OR]
You are right in mentioning that the matching should be case insensitive (the NC flag). The '.*', however, is not neccessary, since in
RewriteCond %{HTTP_USER_AGENT} httrack [NC,OR] the pattern is not anchored anywhere (startĶend of string). The engine will try to match the pattern anywhere in the string. With '^httrack' the pattern is anchored at the beginning, with httrack$ at the end of the string. When you anchored your pattern at the start and end you would need the '.*' if you wanted to match httrack in a string that is not just 'httrack'. Your pattern would need to look like this: '^.*httrack.*$'. Note that this pattern does not make sense, unless you would want to grab the substring before and after the httrack.
To sum up, here is a chart of the four options mentioned above. NA = not anchored; BA = anchored at beginning; EA = anchored at end; MS = modified suggestion.
1. achttrackac (NA: +; BA: -; EA: -; MS: +;)
2. htTracKacac (NA: +; BA: +; EA: -; MS: +;)
3. aacaHttrack (NA: +; BA: -; EA: +; MS: +;)
Note that '.*' will match the '' string, since the quantifier * greedily (rather more than less) matches 0 or more times.
Andreas
I'm trying to ban one site from getting to me. I want to redirect to a page called /robots.php
So I tried this:
rewriteEngine On
rewriteCond {HTTP_REFERER} ^http://(www\.)?domain.com [NC,OR]
RewriteRule ^.*$ /robots.php [L]
but that seems to block everyone. What am I doing wrong?
