Welcome to WebmasterWorld Guest from 54.161.25.142

Forum Moderators: coopster & jatar k & phranque

Message Too Old, No Replies

A Close to perfect .htaccess ban list

     
3:30 am on Oct 23, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Here's the latest rendition of my favorite ongoing artwork....my beloved .htaccess file. I've become quite fond of my little buddy, the .htaccess file, and I love the power it allows me to exclude vermin, pestoids and undesirable entities from my web sites

Gorufu, littleman, Air, SugarKane? You guys see any errors or better ways to do this....anybody got a bot to add....before I stick this in every site I manage.

Feel free to use this on your own site and start blocking bots too.

(the top part is left out)

<Files .htaccess>
deny from all
</Files>
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.]\.your-site.com.* - [F]

3:51 am on Oct 23, 2001 (gmt 0)

10+ Year Member



Nice! Thanks for sharing that really cool info toolman. I can't spot any other bots at the moment.

Sticky

7:24 pm on Oct 23, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Very nice TM. How much speed difference can you notice on each page view?
8:14 pm on Oct 23, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>>How much speed difference can you notice on each page view.

Couldn't say I notice any at all. The part above this though could determine that...if I run everything through the php parser I expect a hit. Usually I run AddHandlers for for ssi's and have never noticed a slow down.

BTW I pieced this together from snippets others posted here on the board.

8:26 pm on Oct 23, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Another one might be

RewriteCond %{HTTP_USER_AGENT} .*almaden.* [OR]
8:51 pm on Oct 23, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I use .htaccess to remap third level domains to various directories based on HTTP_HOST. What happens it two rewritecond's apply to two separate rewrite rules (ie: I place some of these blocking lines above my third level domain remaps in my .htaccess file)?
11:21 am on Oct 24, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Hi Toolman nice compilation of nasty bots! Have you tried sticking the re-writer in httpd.conf? It would run fastest there, although you noted that there was no noticeable speed difference as it is.

Thanks again for sharing it with us!

2:37 pm on Oct 24, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I found another UA for InternetSeer

RewriteCond %{HTTP_USER_AGENT} ^InternetSeer.com [OR]

Not sure what the difference is but this one is the one that comes by every fifteen minutes as my competition tries to fool me into thinking I have more traffic than I do. Now it's easily filtered as a 403.

Long live mod_rewrite :)

3:24 pm on Oct 24, 2001 (gmt 0)

10+ Year Member



toolman- have you been looking over my shoulder at 2 am? I thought *I* had some kind of unhealthy fixation with .htaccess. Guess not. And it may even be healthy, after all.

I've been going back and forth from a kind of banbot.cgi that reads a banned.txt file, to just drawing a line in the sand and doing the full-on mod_rewrite at the top level to initiate a trickle down effect on the sub domains I host.

What I've been toying with is a combination of my banned.txt file automatically updating my .htaccess file - using grep to insert/add/delete lines depending on what is in banned.txt. It's pretty easy to update my banned.txt file either by hand or with a little interface program I wrote - but I'm 'grappling with grep' to insert my lines in the correct place in the .htaccess file. I'm in the dark with grep. Grep vexes me. Grep makes my stomach hurt.

Has anyone else considered this, or is it too much work? I thought it would give me some flexibility, and kill two birds with one stone. In fact, at 2 am I think it's a brilliant idea. Then again, I don't get out much.

6:04 am on Oct 30, 2001 (gmt 0)

10+ Year Member



toolman, mind translating that for those of us are mod_rewrite impaired ?
This 243 message thread spans 25 pages: 243
 

Featured Threads

Hot Threads This Week

Hot Threads This Month