Welcome to WebmasterWorld Guest from 54.166.224.46

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Blocking directory index pages via htaccess

     
2:25 am on Oct 4, 2012 (gmt 0)

10+ Year Member



I recently learned that Google was indexing folder URLs on my site (e.g. /shop/) because when you go to these URLs, the server returns a list of files within the folder. I didn't want these types of URLs indexed by Google. So, I added Options -Indexes to my .htaccess file. Since I also have ErrorDocument 403 /404.htm within my .htaccess, these URLs now redirect to my 404 error page. Unfortunately, that caused Access Denied errors to appear within Google webmaster tools. I have a couple of questions:

One, is adding Options -Indexes to .htaccess the best way to deal with these URLs or is there a better option such as somehow causing the URLs to return 404 instead of 403 status codes?

Two, if I leave these URLs as returning 403 status codes, will the Access Denied errors go away within Google webmaster tools?
2:39 am on Oct 4, 2012 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



You cannot have it both ways.
You either allow indexes or you don't.

There are multiple Google forums here at Webmaster World that could more readily address GWT than the Apache Forum.
4:38 am on Oct 4, 2012 (gmt 0)

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



is adding Options -Indexes to .htaccess the best way to deal with these URLs

Yes. Assuming you don't want humans pawing through the files either. Most people in most circumstances turn off auto-indexing.

these URLs now redirect to my 404 error page

Fortunately you are wrong about this. They don't redirect-- which would be calamitous-- they simply get served the same page as a 404. Or, in the case of robots, they don't. As soon as they see the 403, they slink away unsatisfied.

The rest of the question really does belong in a g### forum.
7:06 am on Oct 4, 2012 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



One, is adding Options -Indexes to .htaccess the best way to deal with these URLs or is there a better option such as somehow causing the URLs to return 404 instead of 403 status codes?

Two, if I leave these URLs as returning 403 status codes, will the Access Denied errors go away within Google webmaster tools?


Options -Indexes is a good idea in any case.

since those directories were already indexed it would probably be better to send a 410 Gone response instead of a 403 Forbidden status code.
you can do that in .htaccess using mad-rewrite and the [G] flag

google often requests previously indexed urls for a long time, so as long as you are forbidding those requests GWT is likely to report them as access denied.
an "error" as reported in GWT is not necessarily a bad thing unless you didn't want that "error" to happen.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month