WMT: Blocking directory index pages via htaccess

[webstuck]

I actually posted this here, but a replier suggested I post in a google forum for better help. [webmasterworld.com...]

I recently learned that Google was indexing folder URLs on my site (e.g. /shop/) because when you go to these URLs, the server returns a list of files within the folder. I didn't want these types of URLs indexed by Google. So, I added Options -Indexes to my .htaccess file. Since I also have ErrorDocument 403 /404.htm within my .htaccess, these URLs now redirect to my 404 error page. Unfortunately, that caused Access Denied errors to appear within Google webmaster tools. I have a couple of questions:

One, is adding Options -Indexes to .htaccess the best way to deal with these URLs or is there a better option such as somehow causing the URLs to return 404 instead of 403 status codes?

Two, if I leave these URLs as returning 403 status codes, will the Access Denied errors go away within Google webmaster tools?

[lucy24]

if I leave these URLs as returning 403 status codes, will the Access Denied errors go away within Google webmaster tools?

No, but that's OK. From GWT's point of view, absolutely anything other than a 200 counts as an error. It doesn't automatically mean you did anything wrong or need to change anything. They're just drawing your attention to the situation in case you didn't intend for it to happen.

In some cases, a redirect may be appropriate. But only do it if you think that humans trying for /directory/ are looking for content that lives on some specific other page-- /directory/starthere.html, let's say.

I answered the Apache aspect of the question in the other forum.

[g1smd]

Returning 403 is fine. You're returning the right response.

The WMT report lets you know that Google has seen that response. Since that response is what you wanted, there's nothing else to do.

redirect to my 404 error page.

There should be no redirect. If requesting a folder results in a 301 or 302 response followed by a 403 or 404 response you have a big problem. The 403 or 404 response should be returned at the originally requested URL.