Custom Error Docs Problems

Inconsistent Display of Custom Error Messages

[Odatpup]

Hello all:

I've tried everything, and my shared hosting tech support can't help me. So now I turn to you guys, hoping you can help me straighten this out. :)

My setup:

A master domain and 3 addon/sub domains on a shared hosting plan. Each of the 4 sites/domains has its own htaccess file, because I need to ban slightly different things for each site. Like this:

/home/MasterName/public_html/
/home/MasterName/public_html/ADDON_1/
/home/MasterName/public_html/ADDON_2/
/home/MasterName/public_html/ADDON_3/

Each addon is its own separate domain name and site.

My goal: To keep each of the 4 domains *totally* independent of each other: to not have visitors ever be aware that these domains are in any way related. This is why I don't want default error messages appearing: they sometimes contain a reference to the MASTER domain name, which I do not want them to see. I know it's picky, but that's how I want it to be.

Each of the 4 domains has Custom Error Documents in its htaccess file (403, 404, 500, and 410); but note that all the Custom Error pages are exactly the same/identical for each domain. Which really irritates me.

I have the Error Document directives at the very top of each individual htaccess file and I am sure the format is correct. This has been verified repeatedly.

Still, here is a pared-down anonymized version of each htacccess file, in case someone sees a problem. All 4 htaccess files follow this basic structure, and things are being banned appropriately, with the exception of the display of default 403s and 500s, as I explained:

ErrorDocument 500 /500.shtml
ErrorDocument 403 /403.shtml
ErrorDocument 404 /404.shtml
ErrorDocument 410 /410.shtml

<Files .htaccess>
order allow,deny
deny from all
</Files>

SetEnvIf Referer ^http://.*\spamsite\.com ban

<Files 403.shtml>
order allow,deny
allow from all
</Files>

deny from env=ban
deny from 4.
deny from 13.

RewriteEngine on
# Check for HTTP/1.1 request Host header 
RewriteCond %{HTTP_HOST} . 
# Return 410 for HTTP/1.1 requests ADDON_1's archives
RewriteRule ^archives/ - [G]

# Ban no ref and no UA 
RewriteCond %{HTTP_REFERER} ^-?$ 
RewriteCond %{HTTP_USER_AGENT} ^-?$ 
RewriteRule!^403\.shtml$ - [F,L] 

# Ban no ref and fake UA (contains Mozilla/x.xx only) 
RewriteCond %{HTTP_REFERER} ^-?$ 
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9]{1,2}$ 
RewriteRule!^403\.shtml$ - [F,L] 

# uninvited
RewriteCond %{HTTP_USER_AGENT} anything [NC,OR]
RewriteCond %{HTTP_USER_AGENT} something [NC]
RewriteRule!^403\.shtml$ - [F,L] 

# crap 
RewriteCond %{HTTP_USER_AGENT} ^(curl¦Dart.?Communications¦Enfish¦htdig¦Java¦larbin) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} (FrontPage¦Indy.?Library¦RPT\-HTTPClient) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^(libwww¦lwp¦PHP¦Python¦www\.thatrobotsite\.com¦webbandit¦Wget¦Zeus) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦Internet¦URL¦WebDAV¦Foundation).(Access¦Explorer¦Control¦MiniRedir¦Class) [NC]
RewriteRule!^403\.shtml$ - [F,L]

# spam 
RewriteCond %{REQUEST_URI} (mail.?form¦form¦form.?mail¦mail¦mailto)\.(cgi¦exe¦pl)$ [NC,OR] 
# MSOffice 
RewriteCond %{REQUEST_URI} ^/(MSOffice¦_vti) [NC]
RewriteRule!^403\.shtml$ - [F,L]

# annoyances
RewriteCond %{HTTP_USER_AGENT} abc [NC,OR]
RewriteCond %{HTTP_USER_AGENT} abcd [NC,OR]
RewriteCond %{HTTP_USER_AGENT} abcde [NC]
RewriteRule!^403\.shtml$ - [F,L]

# no hotlink
RewriteCond %{HTTP_REFERER}!^http://www.myMASTERdomain.com$ [NC]
RewriteRule .*\.(jpg¦jpeg¦gif¦png¦bmpjpg¦jpeg¦gif¦png¦bmp)$ - [F,NC,L]

Question: Do I need to have the list of Custom Error Docs at the top of EACH of my htaccess files? Or should I only need the list at the top of the MASTER domain htaccess file?

Could that be causing Apache to sometimes display the 'too-revealing' default messages for 403s and 500s?

Note that I also have SOME of the same things (spambots, etc.) banned in the MASTERDOMAIN htaccess file as are banned in the ADDON domains' htaccess files. In other words, there is some overlap.

Does that necessarily create a "conflict" that could be causing the inconsistent 403 message (and sometimes 500 message) display problem? Would it be better to "synch" up all 4 htaccess files? Or only use the ONE htacess file at the top level for the MASTER domain and let those directives (hopefully!) affect all the addons/subs below?

I read an article about the "walk" and how htaccess directives are MERGED, but I still can't see what's causing this inconsistency with my Custom Error Documents' display.

Thanks very much in advance for any help. I truly appreciate it. This has been a problem or me now for a couple of years! Finally trying to get to the bottom of it. I'm new to all this so explain like you would to maybe a 10 year old. :) Please.

[jdMorgan]

I've had problems in the past trying to use more that one "Order" directive in one .htaccess file. If I understood the action you want, the following might work better:

SetEnvIf Referer ^http://.*\spamsite\.com ban
Order Allow,Deny
Allow from all
Deny from env=ban
Deny from 4.
Deny from 13.
 
<Files .htaccess>
Deny from all
</Files>
 
<FilesMatch "^(40[34¦410¦500)\.shtml$">
Allow from all
</Files>

Also, not that [L] is redundant when used with [F] or [G], and that you could save some coding by adding the rule:

RewriteRule ^(40[34¦410¦500)\.shtml$ - [L]

ahead of any other rules that produce 403 errors, might invoke a 404 or 410 response, or could cause a 500 error. The effect is to exit mod_rewrite on this pass so that the ErrorDocuments can be served for any request (even from a bad guy).

I don't see any thing else wrong. But I'm rather bleary-eyed, and so may have missed something.

Replace the broken pipe "¦" characters above with solid pipe characters before use. Posting on this forum modifies them.

Jim

[Odatpup]

Jim, THANK YOU so much for the reply.

I will try implenting what you have suggested and post back here either way. I'm very new and understand little, but with this forum's help I have a pretty-much-working htaccess file, which amazes me.

Again, I really appreciate you taking the time to try to help.

:)

[Odatpup]

Hi Jim:

I tried everything; it's not working. I am still not getting any of my CUSTOM ERROR pages when I should be. It still shows the default error pages.

Here's what I did. I stripped down the htaccess file coding for each of the other three domains (MASTERDOMAIN, ADDON_2, and ADD_ON3) to the basics, i.e., nothing but the following in each of those domains' .htaccess files (trying to keep it simple 'til things work):

ErrorDocument 500 /500.shtml
ErrorDocument 403 /403.shtml
ErrorDocument 404 /404.shtml
RewriteEngine on
# No Hotlinking
RewriteCond %{HTTP_REFERER}!^http://addon_2.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_2.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_2.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_2.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_3.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_3.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_3..com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_3.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://masterdomain.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://masterdomain.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_1.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_1.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_1.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_1.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_2.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_2.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_2.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_2.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_3.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_3.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_3.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_3.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.masterdomain.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.masterdomain.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_1.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_1.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_1.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_1.masterdomain.com$ [NC]
RewriteRule .*\.(jpg¦jpeg¦gif¦png¦bmpjpg¦jpeg¦gif¦png¦bmp)$ - [F,NC,L]

Next, I changed the .htaccess file for ADDON_1.com to this:

ErrorDocument 500 /500.shtml
ErrorDocument 403 /403.shtml
ErrorDocument 404 /404.shtml
ErrorDocument 410 /410.shtml
Redirect gone /nothing.html
RedirectMatch gone /archives/*
SetEnvIf Referer ^http://.*\somesite\.com ban
SetEnvIf Referer ^http://.*\someothersite\.com ban
SetEnvIf Referer ^http://.*\blahblah\.fr ban
SetEnvIf Referer ^http://.*\blehbleh\.nl ban
order allow,deny
allow from all
deny from env=ban
deny from 4.
deny from 13.
deny from 18.
deny from 70.84.
deny from 70.85.
deny from 80.
deny from 81.
deny from 82.
deny from 83.
deny from 84.
deny from 202.
deny from 203.
deny from 205.147.9.
deny from 205.180. 
deny from fastres.net
deny from crawler918.com
deny from .ch
deny from .cn
<Files .htaccess>
deny from all
</Files>
<FilesMatch "^(40[34¦410¦500)\.shtml$">
Allow from all
</Files>
RewriteEngine on
# Ban no ref and no UA 
RewriteCond %{HTTP_REFERER} ^-?$ 
RewriteCond %{HTTP_USER_AGENT} ^-?$ 
RewriteRule!^403\.shtml$ - [F] 
# Ban no ref and fake UA (contains Mozilla/x.xx only) 
RewriteCond %{HTTP_REFERER} ^-?$ 
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9]{1,2}$ 
RewriteRule!^403\.shtml$ - [F] 
# uninvited
RewriteCond %{HTTP_USER_AGENT} blahblah [NC,OR]
RewriteCond %{HTTP_USER_AGENT} somethingelse [NC]
RewriteRule!^403\.shtml$ - [F] 
# junk 
RewriteCond %{HTTP_USER_AGENT} ^(curl¦Dart.?Communications¦Enfish¦htdig¦Java¦larbin) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} (FrontPage¦Indy.?Library¦RPT\-HTTPClient) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^(libwww¦lwp¦PHP¦Python¦www\.thatrobotsite\.com¦webbandit¦Wget¦Zeus) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦Internet¦URL¦WebDAV¦Foundation).(Access¦Explorer¦Control¦MiniRedir¦Class) [NC]
RewriteRule!^403\.shtml$ - [F]
# pink lunch meat 
RewriteCond %{REQUEST_URI} (mail.?form¦form¦form.?mail¦mail¦mailto)\.(cgi¦exe¦pl)$ [NC,OR] 
# MSOffice 
RewriteCond %{REQUEST_URI} ^/(MSOffice¦_vti) [NC]
RewriteRule!^403\.shtml$ - [F]
# various invaders
RewriteCond %{HTTP_USER_AGENT} accoona [NC,OR]
RewriteCond %{HTTP_USER_AGENT} alexa [NC,OR]
RewriteCond %{HTTP_USER_AGENT} almaden [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} Anony [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} archiver [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} atriks [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^[CDEFPRS](Browse¦Eval¦Surf) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} Captor [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} chaos [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} Collector [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^(Crawl_Application¦Lachesis¦Nutscrape) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} costacider [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} crawl [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} curl [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} cydral [NC,OR]
RewriteCond %{HTTP_USER_AGENT} cyveillance [NC,OR]
RewriteCond %{HTTP_USER_AGENT} dailyorbit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Demo¦Full.?Web¦Lite¦Production¦Franklin¦Missauga¦Missigua).?(Bot¦Locat) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} digext [NC,OR]
RewriteCond %{HTTP_USER_AGENT} dig [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ding [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^dts\ agent$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} dts\ agent$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (efp@gmx\.net¦hhjhj@yahoo\.com¦lerly\.net¦mapfeatures\.net¦metacarta\.com) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} fastcrawl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} faxo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} fetch [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} geourl [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} girafa [NC,OR]
RewriteCond %{HTTP_USER_AGENT} grafula [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} grub [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^(Industry¦Internet¦IUFW¦Lincoln¦Missouri¦Program).?(Program¦Explore¦Web¦State¦College¦Shareware) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} libcrawl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} linkwalker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Mac¦Ram¦Educate¦WEP).?(Finder¦Search) [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^(Moz+illa¦MSIE).?[0-9]?.?[0-9]?[0-9]?$ [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/[0-9]\.[0-9][0-9]?.\(compatible[\)\ ] [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Mozilla/4\.0\ \(compatible;\ MSIE\ 5\.0;\ Windows\ NT;\ digext;\ dts\ agent$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} nutch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} prowler [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} turnitin [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} Webpix [NC,OR] 
RewriteCond %{HTTP_USER_AGENT} ^Web(\ Image\ Collector¦\ Sucker¦Auto¦Copier¦Fetch) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Web(Reaper¦Sauger¦site\ eXtractor¦Stripper¦Whacker¦ZIP) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} zeus [NC] 
RewriteRule!^403\.shtml$ - [F]
# No Hotlinking (as above)
RewriteCond %{HTTP_REFERER}!^http://addon_2.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_2.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_2.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_2.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_3.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_3.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_3..com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_3.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://masterdomain.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://masterdomain.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_1.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_1.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_1.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://addon_1.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_2.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_2.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_2.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_2.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_3.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_3.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_3.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_3.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.masterdomain.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.masterdomain.masterdomain.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_1.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_1.com$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_1.masterdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER}!^http://www.addon_1.masterdomain.com$ [NC]
RewriteRule .*\.(jpg¦jpeg¦gif¦png¦bmpjpg¦jpeg¦gif¦png¦bmp)$ - [F,NC,L]

Notes:

1. All the broken pipes were whole.

2. I didn't really understand what you meant in your suggestions about [L] and "saving coding"; re: the

RewriteRule ^(403[34¦410¦500)\.shtml$ - [L]

part of your post.

Could you show me specifically where/how to place that? As you see, I just deleted the [L] everywhere except at the very end of the code.

3. I have my CUSTOM ERROR docs (500.shtml, 403.shtml, 410.shtml, and 404.shtml, in EACH domains' directory, at the same level as its .htaccess file. Is this by any chance causing a problem?

4. My hosting service refers to my "ADDONs" also as "SUBDOMAINS" in my cpanel. Could this be part of the issue?

5. Yes I am reaching for any possibilites, at this point.

I don't want to be a pest, but if you get time and feel real charitable :-) could you please help me understand what is going wrong here? Like I said, I'm pretty clueless on this stuff.

WHINE: I just want my Custom Error Documents to SHOW, for all 4 domains. I can't believe how difficult this is, for 2 years now, and my host's tech support won't help or doesn't know what's wrong. :-(

Thank you SO much for ANY help. :-) :-)

[jdMorgan]

This is far too much code to review, but I'll comment on one question:

2. I didn't really understand what you meant in your suggestions about [L] and "saving coding"; re: the
RewriteRule ^(403[34¦410¦500)\.shtml$ - [L] part of your post.

Could you show me specifically where/how to place that? As you see, I just deleted the [L] everywhere except at the very end of the code.

Instead of doing this:

# uninvited
RewriteCond %{HTTP_USER_AGENT} blahblah [NC,OR]
RewriteCond %{HTTP_USER_AGENT} somethingelse [NC]
RewriteRule !^403\.shtml$ - [F]
 
# junk
RewriteCond %{HTTP_USER_AGENT} ^(curl¦Dart.?Communications¦Enfish¦htdig¦Java¦larbin) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (FrontPage¦Indy.?Library¦RPT\-HTTPClient) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww¦lwp¦PHP¦Python¦www\.thatrobotsite\.com¦webbandit¦Wget¦Zeus) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦Internet¦URL¦WebDAV¦Foundation).(Access¦Explorer¦Control¦MiniRedir¦Class) [NC]
RewriteRule !^403\.shtml$ - [F]

Where you have "!^403\.shtml$" in every rule, you can do this:

# Skip remaining access control code if request is for any custom error page
# (Exit from mod_rewrite and allow all to see custom error pages)
RewriteRule ^(403[34]¦410¦500)\.shtml$ - [L]
#
# uninvited
RewriteCond %{HTTP_USER_AGENT} blahblah [NC,OR]
RewriteCond %{HTTP_USER_AGENT} somethingelse [NC]
RewriteRule .* - [F]
 
# junk
RewriteCond %{HTTP_USER_AGENT} ^(curl¦Dart.?Communications¦Enfish¦htdig¦Java¦larbin) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (FrontPage¦Indy.?Library¦RPT\-HTTPClient) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(libwww¦lwp¦PHP¦Python¦www\.thatrobotsite\.com¦webbandit¦Wget¦Zeus) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^(Microsoft¦MFC).(Data¦Internet¦URL¦WebDAV¦Foundation).(Access¦Explorer¦Control¦MiniRedir¦Class) [NC]
Rewrite[b]Rule[/b] .* - [F]

I also suggest that you allow universal access to your robots.txt file and p3p policy (if you have one).

Do not delete the [L] flag from your other rules -- It will slow down your server.

Also, be aware that blocking access by REMOTE_HOST (as in your blocks on .ch and .cn) requires that your server send reverse-DNS lookup requests to the DNS system in order to resolve the IP address. This is extremely inefficient. If you must block by 'country' then I suggest looking up the major IP address ranges for that country. Blocking by IP address is much more efficient. However, I have to question banning entire countries just on principle.

Jim

[edited by: jdMorgan at 2:19 pm (utc) on April 26, 2005]

[Odatpup]

Couldn't get any of it to work, but thanks much!

Re: <<I have to question banning entire countries just on principle.>> Obviously I have my reasons, but I just didn't feel they were relevant to my questions, so did not include them and clutter my already-lengthy post. Appreciated your help.

[jdMorgan]

Well hopefully, it wasn't because of my typo (see correction in bold above)...

Jim