Denying access to all files except robots.txt

[guillermo5000]

I have used .htaccess for some time to deny access, re-direct, and rewrite but I understand there is a way to allow bots access to the robots.txt file without allowing access to other files. In other words, to let everyone access the robots.txt file regardless of other .htaccess commands. If this is true, how is it done. Thank you!

[jdMorgan]

The simplest way is to place the line

RewriteRule ^robots\.txt$ - [L]

above all the other RewriteRules in your .htaccess file.

It says, "If robots.txt is requested, leave the URL unchanged and stop processing mod_rewrite directives." The result is that no further RewriteRules are processed when robots.txt is requested -- mod_rewrite quits, and the file is served.

I use the same approach to allow universal access to robots.txt and my custom 403-Forbidden error page.

Jim

[guillermo5000]

Ah! Thank you!

[guillermo5000]

Something else just occurred to me. I have some deny commands before my Rewrites so the denied IP's will still be disallowed from my robots.txt and error pages. Any solution to that besides changing the deny's to rewrites?

Thanks all!~

[jdMorgan]

For mod_access, you can use mod_setenvif to allow access to robots.txt and a few other resources.

# Block IPs from accessing all but robots.txt and custom 403 page
SetEnvIf Request_URI "(403\.html¦robots\.txt)$" alw
<Files *>
Order Deny,Allow
Deny from 192.168.0.1
Deny from 127.0.0.3
Allow from env=alw
</Files>

Note that the order of module execution in .htaccess is *not* controlled by the order of the directives in the file. In most configurations, mod_access is processed before mod_rewrite. The order of module execution is controlled by the server configuration at start-up. Only directives in the same module are processed in the order you specify.

Jim