mod_rewrite bad robot blocking

Creating a few exceptions

[dorward]

I have a longish mod_rewrite section in my Apache config, partially gleaned from here, partially from other sites, and partially from adding bots I've spotted in my logs.

It looks something like this:

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_REFERER} iaea\.org [OR] # spambot
RewriteCond %{HTTP_USER_AGENT} "DTS Agent" [OR] # OD
RewriteCond %{HTTP_USER_AGENT} "Fetch API Request" [OR] # OD
RewriteCond %{HTTP_USER_AGENT} "Indy Library" [OR] # spambot
...
RewriteCond %{REMOTE_ADDR} ^12\.148\.209\.(19[2-9]¦2[0-4][0-9]¦25[0-5])$ [OR] # NameProtect spybot
RewriteCond %{REMOTE_ADDR} ^64\.140\.49\.6([6-9])$ # Turnitin spybot
RewriteRule .* - [F,L]

Now, I believe that should throw up Forbidden errors to any of the bots on the list trying to access any page. I'd like to be able to allow them access to one or two areas of the site (notably robots.txt).

How can I modify the mod_rewrite rules in this way?

[hakre]

hi dorward, welcome to webmasterworld [webmasterworld.com].

after the RewriteBase / line, insert rewrite rules which will allow a specific part of your site to be accessed by any client. use the [L] flag on it, and other rules won't be checked afterwards. i think this might be the fastest solution for some pages / files / sections.

- hakre