Is a WMT 403 "error" hurting the ability to crawl my site ?

[doughayman]

In WMT, I have a website, that is yielding a 403 error (forbidden) for a subdirectory, under my main domain:

www.example.com/subdir/

My webserver has the "Disable Directory Listings" feature enabled, which is why this error results. I do not like to give people access to my directories carte-blanche, so that is why I have this feature enabled.

Question: From Google's perspective, since this error is raised, am I hurting my ability at to get crawled and indexed, by having this webserver feature enabled ? Has anyone had adverse effects as a result of enabling such a feature ?

[g1smd]

They are merely telling you what they found.

If the response is what you expected for that URL, there is nothing for you to fix.

They have no idea whether that response is correct, so they let you know so you are aware of what they found.

[jdMorgan]

The question is really "Why is someone linking to that URL?"

Make sure that this link to the directory is not on your site.

Jim

[doughayman]

The link to the directory is definitely NOT on my site; that I've checked. However, each sub-directory has a unique index page, so I am going to set up my webserver administratively, to redirect any of these sub-directory queries to map to the appropriate index page. I think that will get rid of this problem.

[g1smd]

*** to redirect any of these sub-directory queries to map to the appropriate index page ***

"Folder name with trailing slash" *is* the canonical URL for the index page.

So, do you want to block it, or serve some content?

If the latter, link to slash-ended URLs from within your site, use the DirectoryIndex directive to get the right file displayed for that URL request, and redirect URL requests that include the index filename to the canonical form without the name included.

[doughayman]

g1smd,

Actually, I want to do both. I don't want anybody "browsing my directories" because I have files in there that don't need to be seen by others, and aren't served by my webserver. However, these subidrectory names are part of valud URL's that I do want to serve up, so if someone browses to my /subdirN/, I think it would be appropriate to redirect to the subdirN main page, in this case.

I must note that I am using an old webserver for this domain (O'Reailly & Associates Website Professional), and that I have about a dozen or so subdirectory-based websites (an old architecture) under this webserver. Each index file for each subdirectory-based website has a unique name, so I cannot use the DirectoryIndex directive carte blanche to cover all subdirectories, without doing some major re-organization first.

My webserver does support redirects, so that a:

www.example.com/SubDir1/ can be redirected to:

www.example.com/SubDir1/SpecificIndexPageName.htm

That shouldn't cause any canonical issues, should it ?

[jdMorgan]

Avoid redirects when you can. They cause an "extra step" for browsers and 'bots alike, and therefore slow things down. Using a redirect for commonly-requested URLs is like paying the post office to forward all mail forever after you've moved, instead of telling your correspondents that you've got a new address -- all of your mail will be delayed by the "extra step."

Assuming that this server is Apache-compatible, you can place a DirectoryIndex directive in a unique .htaccess file in each "Web site's" subdirectory, and define the unique index page name using that directive.

DirectoryIndex is the correct solution, and the one that won't normally cause any other problems.

Jim

[edited by: jdMorgan at 9:02 pm (utc) on June 13, 2009]

[doughayman]

Thanks for the advice, Jim. Unfortunately, as mentioned above, this is an old PC-based webserver, that is NOT compatible with Apache. I don't have DirectoryIndex directive capability, nor Mod ReWrite capability. I don't see any other solution here for me, unless you can think of one.

[dstiles]

If it's IIS then you can set its behaviour in IIS Manager. You can also specify that a site does not allow directory listing. The following works on IIS 5 and 6. Not sure about IIS 4 or 3.

I disallow directory listings (Home Directory tab, Directory Browsing), set the default page in the Documents tab to (in my case) index.asp. To avoid an error page if someone tries to download an index page from (say) a pics folder (eg http://example.com/pics/) I drop a dummy index.asp into the folder, generally redirecting to the home page (Response.Redirect "/").

I haven't found a way of getting IIS to do a 301 to anything other than a subdomain (eg http://example.com) but you can use various Response codes to return any code you like - 301, 404, 403 etc - from within the (eg) index.asp page: example:

Response.Status = "404 Not Found"
Response.End

Hope this helps.

[doughayman]

Thanks, DStiles....many ways to skin a cat. No, I don't use IIS; use O'Reilly and Associates Website Professional (ever heard of it ?). Still ticking, after all these years, but has some limitations, over other mainstream webserver technologies.

[dstiles]

Used it once about 12 years ago. Can't recall if it was on MS or Linux.

The problem with old (ie earlier than last month!) browsers is vulnerability. Although it's probably unlikely anyone would mount a serious attack on such a low-profile server.