Forum Moderators: phranque
What I am doing is redirecting non www pages to the www version and that works fine. I also have a custom error page that is working fine. (All nonexistent pages return a 404 except the custom page itself - that returns a 200 if you go directly to it. I'm assuming that's fine. Maybe not?)
The problem occurs when someone enters a url without the www, and the url does not exist, and the url is in my cgi-bin.
What happens is the nonexistent url returns a 301 and the target location is my custom 404 page that then returns a 200.
Every other nonexistent url without the www returns a 301 and the target location is the url with the www then that returns a 404 which seems correct. Only nonexistent urls in the cgi-bin end up at 200 and only if they do not have the www.
So if that made any sense, it might not hehe :) where do I start to find out how to fix it?
Thanks in advance.
As an example, consider that many sites use their home page as the custom 404 error document -- a practice I discourage for SEO reasons. But nevertheless, a combination custom-404/home page must obviously return a 200-OK when directly requested, or your site would stop working!
However, the behaviour with your cgi-bin requests isn't quite right.
Is this a "shared" cgi-bin directory, containing symbolic links to host-provided scripts available to all users of the server?
If so, it's likely that the directory is not really part of your account's filesystem, but rather, an alias directory that resides in the protected part of the host's filesystem.
And if that's the case, then nothing you can do as a user will affect the behaviour of cgi-bin accesses, because if the host allowed that, then you could affect the behaviour for every other user as well -- A major server security problem.
The above is just speculation based on limited information, but I've had the same thing happen to me. My fix was to copy or symlink the scripts into my own filespace, use robots.txt to Disallow: /cgi-bin from being spidered, and hope that no-one linked to anything in that shared cgi-bin directory.
Jim
it MUST return a 200-OK if it exists.
Okay, good. I was wondering about that - so I just deleted the custom 404 and that fixed it for now.
I don't think my cgi-bin is shared, I don't see anything in there that isn't mine but it is outside of my www folder if that makes a difference.
Maybe I'll just wait and put the custom 404 back after everybody stops linking to the nonexistent pages...in 120 years...
Thanks!
