RewriteRule confusion

.htaccess mod rewrite

[keyplyr]

In an earlier thread, it was suggested that all disallowed agents be sent to robots.txt. This was suggested as the last line:

RewriteRule!^robots\.txt$ - [F]

Ever since I've been using it, I've been getting these errors (this one from Google):

[Fri Apr 4 00:15:35 2003] [error] [client 64.68.82.5] File does not exist: /www/a/apassion4jaz/htdocs/index.htmlrobots.txt

So what's wrong?

[jdMorgan]

keyplr,

The rule you posted does not send anyone to robots.txt...

RewriteRule !^robots\.txt$ - [F]

It says "If the requested file is not robots.txt, do not redirect, but return a 403-Forbidden status.

Parsing it out, "!^robots\.txt" is means not robots.txt, "-" means redirect to (nowhere), and [F] means to return 403-Forbidden status code.

What this does is deny all resources except robots.txt (effects possibly modified by preceding RewriteConds). So unless your code is missing the space just before the "!", I think your problem is elswhere.

Jim

[keyplyr]

Thanks Jim, I never quite understood that. So my problem is still 'why are some bots requesting the robots.txt by adding to the end of the index.html' thus getting the error?

Could it be that my syntax is incorrect?

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^EXAMPLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^EXAMPLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^EXAMPLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^EXAMPLE
RewriteRule!^robots\.txt$ - [F]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule!^http://[^/.]\.mydomain.com.* - [F]

thanks

[KonnyQ]

keyplyr,

I have the following lines in my htacess and it works perfect. Not that I would be profi in mod_rewrite but it works. ;)

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} EXAMPLE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EXAMPLE [NC]
RewriteCond %{HTTP_USER_AGENT} EXAMPLE
RewriteRule!^(/403\.html¦/robots\.txt)$ - [F,L]

Not shure but I think this would also work:

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} EXAMPLE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} EXAMPLE [NC]
RewriteCond %{HTTP_USER_AGENT} EXAMPLE
RewriteRule!^(403\.html¦robots\.txt)$ - [F,L]

edit: this was a similar question I think: [webmasterworld.com...]

Konny