.htaccess blocking rules only affect root not sub-directories

What is the correct way to make rules also apply to folders?

[Wizcrafts]

Greetings again!

I have some pretty complex mod-rewrite conditions and rules that ban bad user agents, or blank user agents, and various known server exploits. Lately I have noticed, while reading my raw access logs, that these rules don't seem to be applied to my /blog directory, so I have started duplicating them in that directory (seems like a needless waste). My rules in question don't have a leading slash that would restrict them to files in the root. The only thing that I suspect is limiting the application to the root is this:

Options +FollowSymLinks
RewriteEngine On
RewriteBase /

< snip: numerous conditions and rules >

If I remove the line:
RewriteBase /
will that allow the rules below it to apply to sub-directories?

Example of rules not applied to sub-directories:

RewriteCond %{HTTP_USER_AGENT} [a-z0-9]{15,}(\s.+)? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^[a-zA-Z0-9]{18,}$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^[a-zA-Z0-9]{18,}\s.+ [OR]
RewriteCond %{HTTP_USER_AGENT} (.+)\s+(.+)\s*.? [NC,OR]
RewriteCond %{HTTP_USER_AGENT} [b-df-hj-np-tvwxz]{5,} [NC]
RewriteCond %{HTTP_USER_AGENT} !(list¦of¦acceptable¦words¦in¦user¦agent) [NC]
RewriteRule .* - [F]

Thanks in advance. Wiz

[edited by: Wizcrafts at 7:22 pm (utc) on May 23, 2007]

[jdMorgan]

It depends on how your blog directory is implemented, but the first thing to check is whether you've got RewriteOptions inherit set on that server.

Jim

[Wizcrafts]

It depends on how your blog directory is implemented, but the first thing to check is whether you've got RewriteOptions inherit set on that server.

Thanks for the input Jim. Is that something I can add to my main .htaccess, or is that something that the hosting company has total control over? I can only manage my own shared account.

Wiz

[edited by: Wizcrafts at 1:47 am (utc) on May 24, 2007]

[Wizcrafts]

I tried commenting out RewriteBase / to no avail. Then I added RewriteOptions inherit but nothing changed. Does this indicate that my web host is in control of these parameters and is not allowing them on all virtual hosts? Is there any other workaround I can use in my master .htaccess, or must I duplicate rules in the sub-directory?

The folder structure is a follows:

public_html is the web root where the master .htaccess resides.
/blogs/ is the blog files sub-directory, where I am duplicating some blocking rules in a separate .htaccess placed in that directory. The URL path resembles: example.com/blogs/ . All rules in the master .htaccess work in the root, but not in sub-directories, unless I insert a copy of .htaccess into those directories.

Wiz

[jdMorgan]

You're not going to make me tell you to read the mod_rewrite documentation to find that directive's context, are ya? :)

The host can set this, but if mod_rewrite is enabled, then you can override the host setting in .htaccess.

Put the RewriteOptions inherit in the subdirectory's .htaccess and see if that makes a difference.

Jim

[Wizcrafts]

Jim;
I did put inherit in the .htaccess in the sub-director, but items blocked in the root are permitted in the sub-dir. Here is the top of the directives I am using in the sub-directory:

Options +FollowSymLinks
RewriteEngine On
RewriteOptions inherit
RewriteBase /

Do I have them in the right sequence?

PS: I did read the documentation for RewriteOptions and it was unclear to me if it applied to individual virtual hosts, or was meant for httpd.conf. Mad Cow.

[Wizcrafts]

You can close this up Jim. I resolved the problem with the addition of "RewriteOptions inherit" and corrected an un-obvious forced path in the root .htaccess that caused my tests to fail. It was a ^ before the test filename, as in:

RewriteCond %{REQUEST_URI} ^foobar\.html$

That restricts the path to root files. I removed the ^ and the rule now works in the sub-directory as well...

RewriteCond %{REQUEST_URI} foobar\.html$

Thanks again, Wiz

[jdMorgan]

Cool! I was beginning to seriously wonder what was going on with your server... Sometimes, they just aren't set up correctly, and really weird things can happen -- Like regex not being parsed correctly. Those kinds of things are really tough to figure out, because "nothing makes sense."

In each Apache directive's section, there's a line at the top, like this one from RewriteOptions:

Context: server config, virtual host, directory, .htaccess

So, this says that the directive can be used in httpd.conf, conf.d or other "included" config files at the server config level. Then it can also be used within <VirtualHost> and <Directory> containers -- also in server config files, and in .htaccess files.

The ones where ".htaccess" is missing from the context list can't be used on shared virtual servers, like RewriteMap, RewriteLog, RewriteLock, and RewriteLogLevel... :(

Jim

[Wizcrafts]

Thanks for adding that information Jim. Even after using the inherit directive in the target directory I still had to edit some of my rules to remove leading ^ symbols or / that set the path to the root. The inherit issue did not exist on my previous host's server. Every rule worked in sub-directories from the get go, unless I forces a forward slash. In fact, I had some instances where a rule was being applied to all sub-directories, which was meant for the root only, and I had to add a forward slash ahead of the filename.

I have also found that as web hosts sell huge numbers of accounts for use on every server they seem to care less about individual webmaster's concerns, regarding scripting issues. I had a ceaseless referer log spammer who attacked me with a script run from a Chinese server, 24/7/365. It was all I could do to get a firewall block placed on that IP, because the hit frequency was less than once per second. Eventually they agreed to block it and I can now read my logs without purging thousands of identical meaningless hits.

One learns or else entropy sets in.

Wiz