Mozilla/3.0 (compatible) still able to access site

Forum Moderators: phranque

Message Too Old, No Replies

Mozilla/3.0 (compatible) still able to access site

Even after banning with .htaccess

cybertime

10:27 am on Dec 20, 2004 (gmt 0)

I have the following in my .htacess file - I am trying to ban Mozilla/3.0 (compatible) - and noticed that a user with Mozilla/3.0 (compatible) was able to access my site without getting a 403 reponse:

RewriteCond %{HTTP_USER_AGENT} ^(Moz+illa¦MSIE).?[0-9]?.?[0-9]?[0-9]?$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9][0-9]?.$compatible[$\ ] [NC,OR]

Web log:
68.70.70....[20/Dec/2004:04:12:23GET /contact.htm HTTP/1.0200-Mozilla/3.0 (compatible)
68.70.70....[20/Dec/2004:04:12:23GET /signup.htm HTTP/1.0200-Mozilla/3.0 (compatible)
68.70.70....[20/Dec/2004:04:12:23GET /sitemap.htm HTTP/1.0200-Mozilla/3.0 (compatible)
68.70.70....[20/Dec/2004:04:12:24GET /cafe.htm HTTP/1.0200-Mozilla/3.0 (compatible)
68.70.70....[20/Dec/2004:04:12:25GET /faq.htm HTTP/1.0200-Mozilla/3.0 (compatible)

Did I mess up the code? Please help.

wilderness

3:36 pm on Dec 20, 2004 (gmt 0)

RewriteCond %{HTTP_USER_AGENT} ^(Moz+illa¦MSIE).?[0-9]?.?[0-9]?[0-9]?$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9][0-9]?.$compatible[$\ ] [NC,OR]

I've not been able to comprehend this methods.
Instead. . .I use three of the following lines:

SetEnvIf User-Agent Mozilla$ keep_out
SetEnvIf User-Agent ^Mozilla/4\.0$ keep_out
SetEnvIf User-Agent ^Mozilla/5\.0$ keep_out
SetEnvIf User-Agent compatible)$ keep_out
SetEnvIf User-Agent compatible\;)$ keep_out

cybertime

4:07 pm on Dec 20, 2004 (gmt 0)

Thank you wildernes.

I am getting error 500 - [Mon Dec 20 11:01:37 2004] [alert] [client ......55.94] /home/virtual/site26/fst/var/www/html/.htaccess: SetEnvIf regex could not be compiled.
- after entering the code in my .htaccess:

Below is a copy of the .htaccess:

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

SetEnvIf Request_URI "^(/403.*\.htm?:/robots\.txt)$" allowit
# Block bad-bots using lines written by bad_bot.pl script above

# Ban .htaccess & .htpasswd requests
SetEnvIfNoCase Request_URI \.ht(access¦passwd)$ ban

<Files *>
Order deny,allow
Deny from env=ban
Deny from env=getout
Allow from env=allowit
</Files>

AuthName wwWebmasterWorldebsite.com
AuthUserFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.pwd
AuthGroupFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.grp

RedirectMatch (.*)\.html$ [wwWebmasterWorldebsite.com$1.htm...]

RewriteCond %{QUERY_STRING} ^id=27$
RewriteRule ^about\.cgi /cgi-bin/trap.pl [L]

DirectoryIndex index.htm

Options +FollowSymLinks
RewriteEngine on
RewriteRule (mail.?form¦form¦form.?mail¦mail¦mailto¦sendmail)\.(cgi¦exe¦pl¦asp¦php¦pm)$ /cgi-bin/trap.pl [NC,L]

RewriteCond %{HTTP_METHOD} ^(PUT¦DELETE¦CONNECT¦HEAD¦PUT)$ [OR]
RewriteCond %{REQUEST_URI} ^\.ht
RewriteRule .* - [F]

RewriteEngine on
RewriteRule ^/?http:// - [F]

# Forbid requests for exploits & annoyances
# Bad requests
RewriteCond %{REQUEST_METHOD}!^(GET¦HEAD¦OPTIONS¦POST¦PUT) [NC,OR]
RewriteCond %{THE_REQUEST} ^/?http [NC,OR]
# Various
RewriteCond %{HTTP_USER_AGENT}!EmailProtect [NC]
RewriteCond %{HTTP_USER_AGENT} ^(BlackWidow¦Crescent¦Disco.?¦ExtractorPr¦HTML.?Works¦Franklin.?Locator) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Green\ Research¦Harvest¦HLoader¦http.?generic¦Industry.?Program) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(IUPUI.?Research.?Bot¦Mac.?Finder¦NetZIP¦NICErsPRO¦NPBot¦PlantyNet_WebRobot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Production.?Bot¦Program.?Shareware¦Teleport.?Pro¦TurnitinBot¦TE¦VOBSUB¦VoidEYE) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(WEP.?Search¦Wge¦Wget¦Zeus.?ThemeSite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(Auto¦Bandit¦Cloner¦Capture¦Devil¦dup¦Fetch¦Filter¦Gather¦Go¦Leach¦Mine¦Mirror¦Pix¦QL¦RACE¦Sauger) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(site.?(eXtractor¦Quester)REAPER¦Snake¦snatcher¦ster¦Strip¦Suck¦vac¦walk¦Whacker¦ZIP) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cherry.?picker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦Internet¦URL¦WebDAV¦Foundation).(Access¦Explorer¦Control¦MiniRedir¦Class) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} e?mail.?(collector¦extractor¦magnet¦reaper¦search¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Boston.?Project[NC,OR]
RewriteCond %{HTTP_USER_AGENT} \.\.\.\.\.\..?¦Educate.?Search¦Full.?Web.?Bot¦Indy.?Library¦IUFW.?Web [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Cowbot¦Downloader¦larbin¦NaverRobot¦QuepasaCreep¦Siphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} OmniWeb [NC,OR]
RewriteCond %{HTTP_USER_AGENT} httrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} booch? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web?Con [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web?Copier.? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} zeus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} wget? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} proxy?scan [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (DTS.?Agent¦Email.?Extrac) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (efp@gmx\.net¦statbot@gmail.com¦hhjhj@yahoo\.com¦lerly\.net¦mapfeatures\.net¦metacarta\.com) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^P\.Arthur\ 1\.1$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Miss.*g.*.?Locat.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.?URL.?Control.? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Smart¦Mass)?.?Download [OR]
# Phoney User_Agents used by email harvesters
RewriteCond %{REQUEST_URI} /(admin¦cmd¦httpodbc¦nsiislog¦root¦shell)\.(dll¦exe) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Moz+illa¦MSIE).?[0-9]?.?[0-9]?[0-9]?$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9][0-9]?.$compatible[$\ ] [NC,OR]
RewriteCond %{REQUEST_URI} /sensepost\.exe [NC]
RewriteCond %{REQUEST_URI} ^/default\.(ida¦idq) [NC,OR]
RewriteCond %{REQUEST_URI} ^/.*\.printer$ [NC,OR]
RewriteCond %{REQUEST_URI} (MSOffice/cltreq\.asp¦_vti_bin/owssvr\.dll¦_vti_bin/_vti_aut/fp30reg\.dll¦_mem_bin¦MSADC¦sumthin) [NC,OR]
# RewriteCond %{REQUEST_URI} ~\!\^~\!\^~\!\.html [OR]
RewriteCond %{HTTP_REFERER} q=guestbook [NC,OR]
RewriteCond %{HTTP_REFERER} iaea\.org [NC]
# Above is last condition ^
RewriteRule!^(docs/403\.htm¦robots\.txt¦other-allowed-files) - [F]

# Forbid if blank Referer *and* UA, except for HEAD requests (used by AOL, etc.)
rewritecond %{REQUEST_METHOD}!^HEAD$
rewritecond %{HTTP_REFERER}<->%{HTTP_USER_AGENT} ^<->$
RewriteRule!^403.*\.htm$ - [F]
#
# Forbid if *faked* blank Referer
rewritecond %{HTTP_REFERER}<->%{HTTP_USER_AGENT} ^-<->-$
RewriteRule!^403.*\.htm$ - [F]

# Block libwww-perl except from AltaVista, Inktomi, and IA Archiver
RewriteCond %{HTTP_USER_AGENT} ^libwww-perl/[0-9] [NC]
RewriteCond %{REMOTE_ADDR}!^209\.73\.(1[6-8][0-9]¦19[01])\.
RewriteCond %{REMOTE_ADDR}!^209\.131\.(3[2-9]¦[45][0-9]¦6[0-3])\.
RewriteCond %{REMOTE_ADDR}!^209\.237\.23[2-5]\.
RewriteRule!^403.*\.htm$ - [F]
#
# Block Java and Python URLlib except from Google
RewriteCond %{HTTP_USER_AGENT} ^(Python.urllib¦Java/?[1-9]\.[0-9]) [NC]
RewriteCond %{REMOTE_ADDR}!^216\.239\.(3[2-9]¦[45][0-9]¦6[0-3])\.
RewriteRule!^403.*\.htm$ - [F]

# Websense
RewriteCond %{REMOTE_ADDR} ^63\.212\.171\.(12[89]¦1[3-9][0-9]¦2[0-4][0-9]¦25[0-4])$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.132\.15\.2(4[0-9]¦5[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^64\.156\.198\.(6[89]¦7[4-8]¦8[0-5])$ [OR]
RewriteCond %{REMOTE_ADDR} ^66\.194\.6\. [OR]
RewriteCond %{REMOTE_ADDR} ^69\.67\.32\.1(4[4-9]¦5[0-9])$ [OR]
#
# N2H2 Content Filtering
RewriteCond %{REMOTE_ADDR} ^66\.150\.1(6[89]¦7[01])\. [OR]
RewriteCond %{REMOTE_ADDR} ^206\.129\.[01]\. [OR]
RewriteCond %{REMOTE_ADDR} ^209\.102\.7[67]\. [OR]
#
# Netsweeper via Hamilton Hydro / FibreWired
RewriteCond %{REMOTE_ADDR} ^66\.207\.(9[6-9]¦1[01][0-9]¦12[0-7])\. [OR]
#
# Covenant Eyes
RewriteCond %{REMOTE_ADDR} ^69\.41\.14\.([1-9]?[0-9]¦1[01][0-9]¦12[0-7])$ [OR]
RewriteRule - [F]

# Forbid if UA is a single word - case-insensitive, A-Z only
RewriteCond %{HTTP_USER_AGENT} ^[a-z]+$ [NC]
# Some exemptions though...
RewriteCond %{HTTP_USER_AGENT}!^ColdFusion$ [OR]
RewriteCond %{HTTP_USER_AGENT}!^DeepIndex$ [OR]
RewriteCond %{HTTP_USER_AGENT}!^FavOrg$ [OR]
RewriteCond %{HTTP_USER_AGENT}!^MantraAgent$ [OR]
RewriteCond %{HTTP_USER_AGENT}!^MARTINI$
RewriteRule!^403.*\.htm$ - [F]

RewriteCond %{HTTP_USER_AGENT} ^[a-z0-9]+
RewriteCond %{HTTP_USER_AGENT}!^msnbot
RewriteCond %{HTTP_USER_AGENT}!^contype
RewriteRule!^403.*\.htm$ - [F]

<IfModule mod_php4.c>
php_value auto_prepend_file "/var/www/html/botblocker.php"
</IfModule>

ErrorDocument 301 /error_testing301.htm
ErrorDocument 403 /403error.htm
ErrorDocument 404 /error_testing.htm

Am I missing something?

Thanks again for your help.

jdMorgan

4:35 pm on Dec 20, 2004 (gmt 0)

Yes, you are missing an 'OR' on your entry for sensepost.exe. Therefore, all user agents above that line must have a user-agent string that is equal to both their own AND senspepost or one of the strings that follows it. This is clearly an impossibility, so you have essentially disabled that entire block of code.

One typo can sink you!

As a side note, your trap code at the top with the querystring test for id=27 up at the top won't be run, because it precedes "RewriteEngine on."

Jim

cybertime

5:03 pm on Dec 20, 2004 (gmt 0)

Thank you jdMorgan.

I have made the corrections:

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

DirectoryIndex index.htm

SetEnvIf Request_URI "^(/403.*\.htm?:/robots\.txt)$" allowit
# Block bad-bots using lines written by bad_bot.pl script above

# Ban .htaccess & .htpasswd requests
SetEnvIfNoCase Request_URI \.ht(access¦passwd)$ ban

<Files *>
Order deny,allow
Deny from env=ban
Deny from env=getout
Allow from env=allowit
</Files>

AuthName www.mysite.com
AuthUserFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.pwd
AuthGroupFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.grp

RedirectMatch (.*)\.html$ [mysite.com$1.htm...]

RewriteEngine on
RewriteRule ^/?http:// - [F]

RewriteCond %{QUERY_STRING} ^id=27$
RewriteRule ^about\.cgi /cgi-bin/trap.pl [L]

Options +FollowSymLinks
RewriteEngine on
RewriteRule (mail.?form¦form¦form.?mail¦mail¦mailto¦sendmail)\.(cgi¦exe¦pl¦asp¦php¦pm)$ /cgi-bin/trap.pl [NC,L]

RewriteCond %{HTTP_METHOD} ^(PUT¦DELETE¦CONNECT¦HEAD¦PUT)$ [OR]
RewriteCond %{REQUEST_URI} ^\.ht
RewriteRule .* - [F]

# Forbid requests for exploits & annoyances
# Bad requests
RewriteCond %{REQUEST_METHOD}!^(GET¦HEAD¦OPTIONS¦POST¦PUT) [NC,OR]
RewriteCond %{THE_REQUEST} ^/?http [NC,OR]
# Various
RewriteCond %{HTTP_USER_AGENT}!EmailProtect [NC]
RewriteCond %{HTTP_USER_AGENT} ^(BlackWidow¦Crescent¦Disco.?¦ExtractorPr¦HTML.?Works¦Franklin.?Locator) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Green\ Research¦Harvest¦HLoader¦http.?generic¦Industry.?Program) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(IUPUI.?Research.?Bot¦Mac.?Finder¦NetZIP¦NICErsPRO¦NPBot¦PlantyNet_WebRobot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Production.?Bot¦Program.?Shareware¦Teleport.?Pro¦TurnitinBot¦TE¦VOBSUB¦VoidEYE) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(WEP.?Search¦Wge¦Wget¦Zeus.?ThemeSite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(Auto¦Bandit¦Cloner¦Capture¦Devil¦dup¦Fetch¦Filter¦Gather¦Go¦Leach¦Mine¦Mirror¦Pix¦QL¦RACE¦Sauger) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(site.?(eXtractor¦Quester)REAPER¦Snake¦snatcher¦ster¦Strip¦Suck¦vac¦walk¦Whacker¦ZIP) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cherry.?picker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦Internet¦URL¦WebDAV¦Foundation).(Access¦Explorer¦Control¦MiniRedir¦Class) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} e?mail.?(collector¦extractor¦magnet¦reaper¦search¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Boston.?Project[NC,OR]
RewriteCond %{HTTP_USER_AGENT} \.\.\.\.\.\..?¦Educate.?Search¦Full.?Web.?Bot¦Indy.?Library¦IUFW.?Web [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Cowbot¦Downloader¦larbin¦NaverRobot¦QuepasaCreep¦Siphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} OmniWeb [NC,OR]
RewriteCond %{HTTP_USER_AGENT} httrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} booch? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web?Con [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Web?Copier.? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} zeus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} wget? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} proxy?scan [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (DTS.?Agent¦Email.?Extrac) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (efp@gmx\.net¦statbot@gmail.com¦hhjhj@yahoo\.com¦lerly\.net¦mapfeatures\.net¦metacarta\.com) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^P\.Arthur\ 1\.1$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Miss.*g.*.?Locat.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.?URL.?Control.? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Smart¦Mass)?.?Download [NC,OR]
# Phoney User_Agents used by email harvesters
RewriteCond %{REQUEST_URI} /(admin¦cmd¦httpodbc¦nsiislog¦root¦shell)\.(dll¦exe) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Moz+illa¦MSIE).?[0-9]?.?[0-9]?[0-9]?$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9][0-9]?.$compatible[$\ ] [NC,OR]
RewriteCond %{REQUEST_URI} /sensepost\.exe [NC,OR]
RewriteCond %{REQUEST_URI} ^/default\.(ida¦idq) [NC,OR]
RewriteCond %{REQUEST_URI} ^/.*\.printer$ [NC,OR]
RewriteCond %{REQUEST_URI} (MSOffice/cltreq\.asp¦_vti_bin/owssvr\.dll¦_vti_bin/_vti_aut/fp30reg\.dll¦_mem_bin¦MSADC¦sumthin) [NC,OR]
# RewriteCond %{REQUEST_URI} ~\!\^~\!\^~\!\.html [OR]
RewriteCond %{HTTP_REFERER} q=guestbook [NC,OR]
RewriteCond %{HTTP_REFERER} iaea\.org [NC]
# Above is last condition ^
RewriteRule!^(docs/403\.htm¦robots\.txt¦other-allowed-files) - [F]

RewriteCond %{HTTP_USER_AGENT} ^[a-z0-9]+
RewriteCond %{HTTP_USER_AGENT}!^msnbot
RewriteCond %{HTTP_USER_AGENT}!^contype
RewriteRule!^403.*\.htm$ - [F]

<IfModule mod_php4.c>
php_value auto_prepend_file "/var/www/html/botblocker.php"
</IfModule>

ErrorDocument 301 /error_testing301.htm
ErrorDocument 403 /403error.htm
ErrorDocument 404 /error_testing.htm

jdMorgan

9:15 pm on Dec 20, 2004 (gmt 0)

I strongly recommend that you remove HEAD from this list.

By disallowing HEAD requests, you prevent many caching proxies from checking your pages for updates. If they cannot use HEAD, they will re-fetch the whole page. That will waste your resources.

 
RewriteCond %{HTTP_METHOD} ^(PUT¦DELETE¦CONNECT¦HEAD¦PUT)$ [OR]
RewriteCond %{REQUEST_URI} ^\.ht
RewriteRule .* - [F]

"PUT" also appears twice in your list...

Jim

cybertime

9:48 pm on Dec 20, 2004 (gmt 0)

Thank you for your assitance jdMorgan.

You have been very helpful.

cybertime

9:53 pm on Dec 21, 2004 (gmt 0)

jdMorgan,

After making the correction as advised, I noticed that a user using Mozilla/3.0 (compatible) - 67.68.152....[21/Dec/2004:13:59:00GET /prepaidcellularcards.htm HTTP/1.0200-Mozilla/3.0 (compatible)
67.68.152...[21/Dec/2004:13:59:15GET /phone.htm HTTP/1.0200-Mozilla/3.0 (compatible)
- was still able to access the site without getting a 403 error.

Here is a copy of the .htaccess file:

# -FrontPage-

IndexIgnore .htaccess */.?* *~ *# */HEADER* */README* */_vti*

DirectoryIndex index.htm

SetEnvIf Request_URI "^(/403.*\.htm?:/robots\.txt)$" allowit
# Block bad-bots using lines written by bad_bot.pl script above

# Ban .htaccess & .htpasswd requests
SetEnvIfNoCase Request_URI \.ht(access¦passwd)$ ban

<Files *>
Order deny,allow
Deny from env=ban
Deny from env=getout
Allow from env=allowit
</Files>

AuthName www.example.ca
AuthUserFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.pwd
AuthGroupFile /home/virtual/site26/fst/var/www/html/_vti_pvt/service.grp

RedirectMatch (.*)\.html$ http://www.example.ca$1.htm

RewriteEngine on
RewriteRule ^/?http:// - [F]

Options +FollowSymLinks
RewriteEngine on
RewriteRule (mail.?form¦form¦form.?mail¦mail¦mailto¦sendmail)\.(cgi¦exe¦pl¦asp¦php¦pm)$ /cgi-bin/trap.pl [NC,L]

RewriteCond %{QUERY_STRING} ^id=27$
RewriteRule ^about\.cgi /cgi-bin/trap.pl [L]

RewriteCond %{HTTP_METHOD} ^(PUT¦DELETE¦CONNECT)$ [OR]
RewriteCond %{REQUEST_URI} ^\.ht
RewriteRule .* - [F]

# Restrict HTTP methods
RewriteCond %{REQUEST_METHOD} !^(GET¦OPTIONS¦POST)$
RewriteRule .* - [F]

# Forbid requests for exploits & annoyances
# Bad requests
RewriteCond %{THE_REQUEST} ^/?http [NC,OR]
# Various
RewriteCond %{HTTP_USER_AGENT} !EmailProtect [NC]
RewriteCond %{HTTP_USER_AGENT} ^(BlackWidow¦Crescent¦Disco.?¦ExtractorPr¦HTML.?Works¦Franklin.?Locator) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Green\ Research¦Harvest¦HLoader¦http.?generic¦Industry.?Program) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(IUPUI.?Research.?Bot¦Mac.?Finder¦NetZIP¦NICErsPRO¦NPBot¦PlantyNet_WebRobot) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Production.?Bot¦Program.?Shareware¦Teleport.?Pro¦TurnitinBot¦TE¦VOBSUB¦VoidEYE) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(WEP.?Search¦Wge¦Wget¦Zeus.?ThemeSite) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(Auto¦Bandit¦Copier¦Cloner¦Con¦Capture¦Devil¦dup¦Fetch¦Filter¦Gather¦Go¦Leach¦Mine¦Mirror¦Pix¦QL¦RACE¦Sauger) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web.?(site.?(eXtractor¦Quester)REAPER¦Snake¦snatcher¦ster¦Strip¦Suck¦vac¦walk¦Whacker¦ZIP) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cherry.?picker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦Internet¦URL¦WebDAV¦Foundation).(Access¦Explorer¦Control¦MiniRedir¦Class) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} e?mail.?(collector¦extractor¦magnet¦reaper¦search¦siphon¦sweeper¦harvest¦collect¦wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Boston.?Project[NC,OR]
RewriteCond %{HTTP_USER_AGENT} \.\.\.\.\.\..?¦Educate.?Search¦Full.?Web.?Bot¦Indy.?Library¦IUFW.?Web [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Cowbot¦Downloader¦larbin¦NaverRobot¦QuepasaCreep¦Siphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} OmniWeb [NC,OR]
RewriteCond %{HTTP_USER_AGENT} httrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} booch? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} proxy?scan [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Microsoft.?URL.?Control.? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (DTS.?Agent¦Email.?Extrac) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (efp@gmx\.net¦statbot@gmail.com¦hhjhj@yahoo\.com¦lerly\.net¦mapfeatures\.net¦metacarta\.com) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^P\.Arthur\ 1\.1$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Miss.*g.*.?Locat.* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Smart¦Mass)?.?Download [NC,OR]
# Phoney User_Agents used by email harvesters
RewriteCond %{REQUEST_URI} /(admin¦cmd¦httpodbc¦nsiislog¦root¦shell)\.(dll¦exe) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Moz+illa¦MSIE).?[0-9]?.?[0-9]?[0-9]?$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9][0-9]?.$compatible[$\ ] [NC,OR]
RewriteCond %{REQUEST_URI} /sensepost\.exe [NC,OR]
RewriteCond %{REQUEST_URI} ^/default\.(ida¦idq) [NC,OR]
RewriteCond %{REQUEST_URI} ^/.*\.printer$ [NC,OR]
RewriteCond %{REQUEST_URI} (MSOffice/cltreq\.asp¦_vti_bin/owssvr\.dll¦_vti_bin/_vti_aut/fp30reg\.dll¦_mem_bin¦MSADC¦sumthin) [NC,OR]
RewriteCond %{HTTP_REFERER} q=guestbook [NC,OR]
RewriteCond %{HTTP_REFERER} iaea\.org [NC]
# RewriteCond %{REQUEST_URI} ~\!\^~\!\^~\!\.html [OR]
# Above is last condition ^
RewriteRule !^(docs/403\.htm¦robots\.txt¦other-allowed-files) - [F]

# Forbid if blank Referer *and* UA, except for HEAD requests (used by AOL, etc.)
rewritecond %{REQUEST_METHOD} !^HEAD$
rewritecond %{HTTP_REFERER}<->%{HTTP_USER_AGENT} ^<->$
RewriteRule !^403.*\.htm$ - [F]
#
# Forbid if *faked* blank Referer
rewritecond %{HTTP_REFERER}<->%{HTTP_USER_AGENT} ^-<->-$
RewriteRule !^403.*\.htm$ - [F]

# Block libwww-perl except from AltaVista, Inktomi, and IA Archiver
RewriteCond %{HTTP_USER_AGENT} libwww-perl/[0-9] [NC]
RewriteCond %{REMOTE_ADDR} !^209\.73\.(1[6-8][0-9]¦19[01])\.
RewriteCond %{REMOTE_ADDR} !^209\.131\.(3[2-9]¦[45][0-9]¦6[0-3])\.
RewriteCond %{REMOTE_ADDR} !^209\.237\.23[2-5]\.
RewriteRule !^403.*\.htm$ - [F]
#
# Block Java and Python URLlib except from Google
RewriteCond %{HTTP_USER_AGENT} (Python.urllib¦Java/?[1-9]\.[0-9]) [NC]
RewriteCond %{REMOTE_ADDR} !^216\.239\.(3[2-9]¦[45][0-9]¦6[0-3])\.
RewriteRule !^403.*\.htm$ - [F]

# Forbid if UA is a single word - case-insensitive, A-Z only
RewriteCond %{HTTP_USER_AGENT} ^[a-z]+$ [NC]
# Some exemptions though...
RewriteCond %{HTTP_USER_AGENT} !^ColdFusion$ [OR]
RewriteCond %{HTTP_USER_AGENT} !^DeepIndex$ [OR]
RewriteCond %{HTTP_USER_AGENT} !^FavOrg$ [OR]
RewriteCond %{HTTP_USER_AGENT} !^MantraAgent$ [OR]
RewriteCond %{HTTP_USER_AGENT} !^MARTINI$
RewriteRule !^403.*\.htm$ - [F]

RewriteCond %{HTTP_USER_AGENT} ^[a-z0-9]+
RewriteCond %{HTTP_USER_AGENT} !^msnbot
RewriteCond %{HTTP_USER_AGENT} !^contype
RewriteRule !^403.*\.htm$ - [F]

<IfModule mod_php4.c>
php_value auto_prepend_file "/var/www/html/botblocker.php"
</IfModule>

ErrorDocument 301 /error_testing301.htm
ErrorDocument 403 /403error.htm
ErrorDocument 404 /error_testing.htm

Am I still missing something?

Thanks for your help.

[edited by: jdMorgan at 2:43 pm (utc) on Dec. 22, 2004]
[edit reason] Obscured specifics [/edit]