Got a visit from one of those brainless bots. The ones that come in with a shopping list, and by golly they're going to work through the whole list, no matter what response they get. In full, skipping redundancies:
27.37.7.214 - - [12/Apr/2012:04:03:16 -0700] "GET / HTTP/1.1" 200 1550 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
... /index.htm HTTP/1.1" 404 1442 ...
... /index.html HTTP/1.1" 200 1550 ...
... /index.asp HTTP/1.1" 404 1442 ...
... /index.aspx HTTP/1.1" 404 1442 ...
... /index.php HTTP/1.1" 403 1479 ...
... /default.htm HTTP/1.1" 404 1442 ...
... /default.html HTTP/1.1" 200 1550 ...
... /default.asp HTTP/1.1" 404 1442 ...
... /default.aspx HTTP/1.1" 404 1442 ...
... /default.php HTTP/1.1" 403 1479 ...
... /main.htm HTTP/1.1" 404 1442 ...
... /main.html HTTP/1.1" 200 1550 ...
... /main.asp HTTP/1.1" 404 1442 ...
... /main.aspx HTTP/1.1" 404 1442 ...
27.37.7.214 - - [12/Apr/2012:04:03:24 -0700] "GET /main.php HTTP/1.1" 403 1479 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)"
That is: The very first thing they ask for is the domain name. It is duly served up. Do they go away? No, of course not. Their shopping list comes with fifteen more items-- three possible names multiplied by five extensions. (They don't seem to have heard of .jsp. Huh.) Matter of fact I'm surprised the list wasn't twice as long, allowing for with-and-without-www versions of each.
It looks jazzier in my logs, because it's color-coded by response. 404 for nonexistent file-- ordinarily that would be 14 of the 15 requests. 403 for lockouts-- here, anything ending in php, because I don't have php files, so anyone who asks for them is up to no good. Except in specific, excluded cases.
What stopped me in my tracks was those other 200s. How the ### did they achieve a 200 response for a nonexistent file?
That's where the silver lining comes in. I fine-tooth-combed my htaccess and realized I'd got a couple of key RewriteRules in the wrong place. If people get rewritten to a custom page, Apache never gets a chance to discover that either the user is banned, or their original request doesn't exist. Possibly both.
This time it was the MSIE 6 that did it. With the rule moved down to proper place, they will still get a 200 for nonexistent files-- because I'm not going to bother with !-f and !-d conditions-- but any mod_rewrite-based lockouts will come before the rewrite.
Punch line: They're from China and by all rights should have been banned regardless. But they live in a range too small to bother about (my current cutoff is /13 except when someone truly offends me), unless I wanted an htaccess that's longer than most of my pages themselves.
... and that's why you have to list your RewriteRules in order of severity, from [F] to [G] to [R=301,L], saving the bare [L] for last.
