If you are Gonna use ReWriteRules to Exclude UA's...

You might want to make a small change

[carfac]

Hi:

I have been looking at my logs since I added a bunch of rewrite rules to exclude certain User Agents, and I see they are getting a LOT of 403 errors. I thought there ought to be something I culd do to be a bit more polite..

So, I added hs to the top of the ReWriteConditions and Rules that exclude UA's:

RewriteRule /robots.txt$ /robots.txt [L]

It has to go BEFORE the list of UA's to be excluded. All it does is say if you are going to ask for robots.txt, serve the robots.txt... and then STOP applying rewrite rules (The [L] rule). I figured it might be a bit nicer than just sending a 403.

If you think there is a problem, of if this creates a security hole, let me know!

dave

[bird]

Under some conditions (never figured out all the details), your rule might lead to infinite recursion. If you want to avoid this, just use the "do nothing" rule instead:

RewriteRule /robots.txt$ - [L]

This really does nothing, in terms of rewriting, except that the L flag stops all further rewrite activity. The file itself will be served as expected.

[carfac]

Bird:

Thanks- that was what I was looking for. I am still a bit iffy on writing "ReWrite"... but that (what you wrote) was my goal.

Thank you!

dave