.htaccess ban list

[Red5]

In the thread "A Close to perfect .htaccess ban list" (here: [webmasterworld.com...] ), a solution to banning email harvesters and rogue crawlers is presented.

I already have a .htaccess file containing a rewrite rule to prevent hot linking (as below)

//---------- START ----------
RewriteEngine on
RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://mysite.com.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.mysite.com.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://mysite.co.uk.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.mysite.co.uk.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://mysite.com.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.mysite.com.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://mysite.co.uk.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.mysite.co.uk.*$ [NC]
RewriteRule .*\.(gif¦jpg¦jpeg¦bmp)$ - [F,NC]
//---------- END ----------

Can I simply add in the code for the ban list to the file and have two rewrite rules in the same file, or is it more complicated than that?

Thanks,

Red5

[Red5]

Bump

[jdMorgan]

Red5,

You can have as many rewrites as you like - subject only to server performance impact. I use several hundred.

Note also that you can compress your existing rules...

RewriteCond %{HTTP_REFERER} !^http://mysite.com.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.mysite.com.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://mysite.co.uk.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.mysite.co.uk.*$ [NC]
RewriteRule .*\.(gif¦jpg¦jpeg¦bmp)$ - [F,NC]

Can be written:

RewriteCond %{HTTP_REFERER} !^http://(www\.)?mysite\.(com¦co\.uk) [NC]
RewriteRule \.(gif¦jpg¦jpeg¦bmp)$ - [F,NC]

You must replace the "¦" characters with the solid vertical pipe character from your keyboard - this board modifies these characters in posts for some reason. Note also that literal periods in rewrite patterns should be escaped with a preceding "\". Also, there is no need to precede unanchored patterns with ".*" or to anchor patterns which end with ".*". The code I posted has these changes incorported.

HTH,
Jim

[Red5]

Hi Jim,

Thank you very much for your comprehensive reply.

So, I could have the following in my .htaccess...

//---------- START ----------
ErrorDocument 404 [mysite.com...]

<Files .htaccess>
deny from all
</Files>

RewriteEngine on

RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://(www\.)?mysite\.(com¦co\.uk) [NC]
RewriteRule \.(gif¦jpg¦jpeg¦bmp)$ - [F,NC]
RewriteCond %{HTTP_REFERER}!^http://search.atomz.com.*$ [NC]
RewriteRule .*\.(gif¦jpg¦jpeg¦bmp)$ - [F,NC]

RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule!^http://[^/.]\.mysite.com.* - [F]
//---------- END ----------

Is this valid?