homepage Welcome to WebmasterWorld Guest from 54.227.20.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

    
Blocking directory index pages via htaccess
webstuck

5+ Year Member



 
Msg#: 4503794 posted 2:25 am on Oct 4, 2012 (gmt 0)

I recently learned that Google was indexing folder URLs on my site (e.g. /shop/) because when you go to these URLs, the server returns a list of files within the folder. I didn't want these types of URLs indexed by Google. So, I added Options -Indexes to my .htaccess file. Since I also have ErrorDocument 403 /404.htm within my .htaccess, these URLs now redirect to my 404 error page. Unfortunately, that caused Access Denied errors to appear within Google webmaster tools. I have a couple of questions:

One, is adding Options -Indexes to .htaccess the best way to deal with these URLs or is there a better option such as somehow causing the URLs to return 404 instead of 403 status codes?

Two, if I leave these URLs as returning 403 status codes, will the Access Denied errors go away within Google webmaster tools?

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4503794 posted 2:39 am on Oct 4, 2012 (gmt 0)

You cannot have it both ways.
You either allow indexes or you don't.

There are multiple Google forums here at Webmaster World that could more readily address GWT than the Apache Forum.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4503794 posted 4:38 am on Oct 4, 2012 (gmt 0)

is adding Options -Indexes to .htaccess the best way to deal with these URLs

Yes. Assuming you don't want humans pawing through the files either. Most people in most circumstances turn off auto-indexing.

these URLs now redirect to my 404 error page

Fortunately you are wrong about this. They don't redirect-- which would be calamitous-- they simply get served the same page as a 404. Or, in the case of robots, they don't. As soon as they see the 403, they slink away unsatisfied.

The rest of the question really does belong in a g### forum.

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4503794 posted 7:06 am on Oct 4, 2012 (gmt 0)

One, is adding Options -Indexes to .htaccess the best way to deal with these URLs or is there a better option such as somehow causing the URLs to return 404 instead of 403 status codes?

Two, if I leave these URLs as returning 403 status codes, will the Access Denied errors go away within Google webmaster tools?


Options -Indexes is a good idea in any case.

since those directories were already indexed it would probably be better to send a 410 Gone response instead of a 403 Forbidden status code.
you can do that in .htaccess using mad-rewrite and the [G] flag

google often requests previously indexed urls for a long time, so as long as you are forbidding those requests GWT is likely to report them as access denied.
an "error" as reported in GWT is not necessarily a bad thing unless you didn't want that "error" to happen.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved