return 404 instead of 403 for directories without index files

[jcary85]

Hi,
For security and SEO reasons, we want to return a 404 response insatead of a 403 response for directories with no index file. Is this possible with apache? I've googled this a lot, and i've seen the question a lot, but never an answer.

[jdMorgan]

Sure. Regardless of the directory-depth, if there is no filetype in the requested URI and the URI does not resolve to an existing directory-index file, then internally rewrite to an known-non-existent file to force a 404:

RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^/]+/)*([^./]+)?$ /filepath-which-does-not-exist [L]

Since you did not specify, this code is for use in .htaccess or within a <Directory> container in a config file on Apace 1.x, and assumes that you've already got other working mod_rewrite code in the context in which you intend to use it.

For use in a config file outside of any <Directory> container, add a slash to the beginning of the RewriteRule's pattern after the start-anchor.

For use on Apache 2.x, you may change the rewriterule to

RewriteRule ^([^/]+/)*([^./]+)?$ - [R=404,L]

if you prefer. Retain the RewriteCond as-is.

The check for "no filetype" (i.e. no period in final URL-path-part) improves efficiency, because it avoids the resource-intensive filesystem check for the majority of requests (such as those for images, css and external js files, etc.). The rewritecond won't be processed at all unless the RewriteRule pattern matches.

Note that because of this exclusion, if an index file is explicitly requested this rule won't execute, and that case will be handled by the server's default 404 handling if the requested index file does not exist.

Jim

[jcary85]

Thanks for the response. Unfortunately i can't get it to work.

With this, i still get 403
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^/]+/)*([^./]+)?$ /filepath-which-does-not-exist [L]

With this, i still get 403
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^/]+/)*([^./]+)?$ - [R=404,L]

with this, pages just aren't served site-wide (lloks like .htaccess error)
RewriteRule ^([^/]+/)*([^./]+)?$ - [R=404,L]

I'm adding this to our curent .htaccess at teh top under teh rewriteengine on line and we are on apache 2.2.3.


Thanks for the help

[jdMorgan]

If you are still getting a 403 on Apache 2.2.3 with this code:

RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([^/]+/)*([^./]+)?$ - [R=404,L]

Then it make take a more explicit approach:

RewriteCond %{REQUEST_FILENAME} ^([^/]+/)*(index|default|home)\.[a-z0-9]$
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^/]+/)*([^./]+)?$ - [R=404,L]

You may need to adapt the first rewritecond to accomodate all possible index page names on your server. Keep in mind that "REQUEST_FILENAME" refers to a filepath and not to the requested URL-path. The request may be for the URL-path "/dir/" but resolve to the file "/dir/index.php3", for example. That is why the RewriteCond in this code is *not* redundant with the RewriteRule pattern -- They are looking at different things.

Jim

[jcary85]

Jim,
Appreciate all teh help, but i still can't get it to work. Our Index files are only index.htm index.html and index.php and the code you provided is still getting 404. Maybe there's an easier way. is there a way to return 404 for ALL 403s? Maybe there is a way to write a RewriteCond for if returning 403, instead return a 404? We have no need for returning a 403 on our site.

[jcary85]

I'm not sure i was entirely clear with my original questions. We want a 404 to be returned on this condition:

user tried to access directory root but directory root has no index file. For example, [domain.com...] -- this has no file name, and let's say this directory has no index file. We have Indexes turned off, so the user gets a 403. We want them to get a 404 instead.

[jdMorgan]

The code I posted will entirely replace "Options -Indexes" and will require that you turn this option back on.

Jim

[jcary85]

Jim,
I appreciate all the help, but i still can't get this to work with Indexes turned on. I've tried every combination of rules you have posted and now just get the directory index listing page every time.

[jdMorgan]

Where in the server filepaths is this code located?

Jim