if /muieblackcat crosses your path

when not to block .php requests

[lucy24]

There have been a couple of recent threads over in the Robot Identification forum about evil robots who request every possible permutation of myadmin.php, optionally preceded by a request for /muieblackcat. The latter apparently means that one of the robot's friends has been there before.

After a while this led to the generic advice: if you don't actually have any php on your site, throw a quick [F] at anyone from anywhere who requests anything in \.php$. Problem solved.

Unless...

History:
I noticed that a couple of local .htaccess files for +Indexes had stopped working. My host was doing some work recently, so I checked with them first. Nope, 'tain't us, check those big complicated htaccess files at the top level.

Lovely. This is where g1 gets to gloat and say I told you not to combine mod_rewrite and mod_alias in a single .htaccess.

Well, maybe not. The culprit turned out to be the single line

RewriteRule \.php$ - [F]

Haven't heard back from the host yet, but I have to guess there is some kind of php involvement in those auto-generated indexes. (I'm hoping they come through with an ! escape clause so I can allow this php and none other. The logs don't give any useful hints.) So by blocking all php requests, I'd disabled the feature all the way up the line. A local +Indexes can override a global -Indexes, but it can't open a side door when the front door is locked.

Lesson:
You can unconditionally stop a whole class of malefactors at the gate, or you can have auto-indexes, but you can't have both. Drat.

[phranque]

did you get a message in your server error log identifying the .php script used for creating the index on the fly?
if you can identify the script you should be able to add a RewriteCond to exclude that script from the RewriteRule blocking php scripts in general.

[lucy24]

No such luck. Host says it's all done by apache. And the error logs just have a single 403 responding to my request for the directory, replaced by a 200 when I get rid of the .php line.

Identical behavior in two different domains. They live in the same general area, but they've each got their own htaccess. It's the minimalist type of auto-index: HTML 3.2 (!), <pre> for pseudo-table format, sort options for each column. (Which isn't really a column. This is a mystery to me.) Not the Tomcat kind of auto-index that's an actual table.

[g1smd]

Add a RewriteCond looking at THE_REQUEST so that you only block external .php URL requests.

By doing that, it is irrelevant what the files and scripts are called.

URLs are used "out there on the web" and paths and files are used "here inside the server". They are not at all the same thing, merely "related".

[phranque]

what g1smd said.
it's most likely that REQUEST_FILENAME would show the php script used for generating the index directories but THE_REQUEST should always show the requested directory (i.e. trailing slash url)

[lucy24]

it's most likely that REQUEST_FILENAME would show the php script used for generating the index directories but THE_REQUEST should always show the requested directory (i.e. trailing slash url)

But I'm not blocking requests for directories, I'm blocking requests for php files :(

[g1smd]

I'm blocking requests for php files

You want to block all .php files? You want to block them even when the URL request does not contain ".php" but the server uses a .php file to process the request?

OR

You want to block all .php URL requests?

The latter is probably the right course of action.

[lucy24]

The latter, I think, even if the host won't admit to any php involvement. I'd always assumed that if anyone, anywhere, asks for something, it will show up in the logs (sometimes either Access or Error but not both). And, conversely, that if it isn't in the logs it can't be affected by a site-specific .htaccess. But apparently this is wrong, since there appears to be an invisible php script interfering with auto-indexing.

[g1smd]

A preceding RewriteCond looking at THE_REQUEST and checking for .php somewhere in there will solve the problem.

[lucy24]

:: mopping brow ::

RewriteCond %{THE_REQUEST} ^GET\ http
RewriteRule \.php$ - [F]

>> auto-indexing happens as desired

RewriteCond %{THE_REQUEST} ^GET
RewriteRule \.php$ - [F]

>> auto-indexing doesn't happen (because everything starts with GET)

Oh, and the server gets very very unhappy if the last thing in a RewriteCond is an escaped space. (I experimented.) So unhappy, in fact, that it won't cough up the custom 500 page.

I got the LiveHeaders extension and did it in Firefox but they had nothing to say about php requests going on in the background. Just 200 or 403. Useful to track the redirects, though. The whole auto-indexing exercise also confirms that I don't have one of those nasty ISPs that do remote caching and refuse to serve up a fresh copy of the page if you've been there within human memory. (One local ISP is notorious for this. Sometimes the cached version will be a blog comment complete with previous user's real name, e-mail and so on.)

[g1smd]

You should be checking that

.php

is in THE_REQUEST.

[lucy24]

RewriteCond %{THE_REQUEST} GET\ [^\s.]+\.php\ HTTP/1\.[01]
or simply
RewriteCond %{THE_REQUEST} \.php

>> auto-indexing works

RewriteCond %{THE_REQUEST} !GET\ [^\s.]+\.php\ HTTP/1\.[01]
or simply
RewriteCond %{THE_REQUEST} !\.php

>> auto-indexing doesn't happen

Can someone explain in words of two syllables what this actually means?

:(

[phranque]

it means if the visitor's user agent requests a url containing ".php" then Forbid that request.
therefore auto-indexing works since the .php never occurs in the user's request for a directory listing.