| 6:41 pm on Mar 13, 2010 (gmt 0)|
I would think that time will straighten this out - but you could also use a reconsideration request or post over in Google's webmaster forums to bring some attention to the problem.
| 6:53 pm on Mar 13, 2010 (gmt 0)|
Thanks very much! Your reply should help to keep me calm for a few days, if nothing changes I will try the routes you suggested.
| 7:09 pm on Mar 13, 2010 (gmt 0)|
Even easier than tedster's suggestion and without worrying about bringing too much attention to it is to simply resubmit the URL to make sure GBot drops by on the next run... It's something I've done before when I'm on a 'wider' crawling schedule and have made a change to a page I want to make sure gets found ASAP or want a page I had been serving a 403 to all bots :) reincluded in the index. I'd just drop the URL in the 'add' box on Google and not even worry about it.
They don't (in my recent experience) crawl pages delivering error status codes nearly as often as they do regular pages and will even basically 'cut off' pages serving error codes from the crawls in some cases. Dropping the URL in the 'add' box on Google seems to trigger an earlier recrawl. (It's not something I've really studied to say for sure, but I've done what I suggested on occasion and it does seem to get the URL recrawled faster.)
When you submit a URL they que it into the crawl, and IMO that's what you need more than anything really.
[edited by: TheMadScientist at 7:15 pm (utc) on Mar 13, 2010]
| 7:14 pm on Mar 13, 2010 (gmt 0)|
This also makes me wonder if your server actually returned a correct http status code - rather than just serving up an error message page, but with a 200 OK status.
| 7:20 pm on Mar 13, 2010 (gmt 0)|
Good point tedster...
It could have been a 302 redirect too if somehow there was a custom error page being served. Of course, normally AFAIK 302s don't happen on Internal Server Errors, but I suppose it's possible. I would suggest setting up a test directory and making an .htaccess file error so you can check the actual status code for the error message served and make sure you are getting the correct status code with a header check like the one here in your control panel.
| 7:27 pm on Mar 13, 2010 (gmt 0)|
I will have a look at my logs tomorrow, guess I should have done that earlier. It might well be what you suspect tedster. The error page was part of the software package and unfortunately that is one of the things I never had a closer look at.
I don't think the crawling is a problem, the Google cache gets renewed almost every day, but for some reason it's only the cache, not the serps.
| 7:39 pm on Mar 13, 2010 (gmt 0)|
Couldn't really wait until tomorrow... according to the logs there are no errors, just status 200.
What are my chances in this case that this will just 'go away'?
| 9:05 pm on Mar 13, 2010 (gmt 0)|
|...according to the logs there are no errors, just status 200. |
But if your error page returned a 200 when you were down, that would have created a problem. You want your error page to return a 404.
Try entering a url for a page on your domain that doesn't exist and make sure that it is returning a 404.
...should show a 404 status.
That doesn't tell you what your server was returning for your home page when your site was down, but it should give you an idea whether you still have problems.
| 12:10 am on Mar 14, 2010 (gmt 0)|
I'm confused. My htaccess file has the following rewrite rule:
I don't see any errors in my logs that correspond to the time the site was down. What the serps show is the title and content of the custom 500 error page. I found these instances in the logs that correspond to this:
|ErrorDocument 500 /500.shtml |
|"GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" |
I don't know what this means exactly, but this probably caused the problem.
|"GET /500.shtml HTTP/1.1" 200 538 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" |
The 403 / 404 pages I set up myself and they don't return 200's.
I have deleted the 500 page for now.
| 12:32 am on Mar 14, 2010 (gmt 0)|
|You want your error page to return a 404. |
Actually, you want the Error Page to send the correct Error Status Code, which in this case is a 500 Internal Server Error, not a 404. If you really want a custom error page for the 500, then you might try changing it to PHP and serving the header directly from the page to make sure User-Agents get the correct status any time they request the page. Personally, I wouldn't worry about it, because if there is really an internal server error, then your server probably can't serve the custom page, which may be why you were getting the odd status code, but if somehow it does, then you could probably make it work by serving the correct status code from the error page itself, so when a Bot requests the page they would get a 302 redirect (which is recommended against), but would be followed by a 500 Internal Server Error on the landing page, which I would include a noindex meta tag on personally.
| 12:48 am on Mar 14, 2010 (gmt 0)|
Yeah, what he said. It looks like your server (damn that MS) has the default error page set-up for 500 that goes like this:
302 Temporary Redirect >> 200 OK.
That's why your error page got indexed. As long as the redirect is a 302 status, Google will index the content of the target page with the address of the original page. Still, this should all be sorted out quite soon for you. Now you just want to prevent any re-occurrence.
Actually, this set-up might even have saved you some entry traffic. There's a chance your rankings would have vanished completely for a while otherwise, as Google does not want to send traffic to a server that isn't responding. But the minute they got a good response again, things would have straightened out and you would not ever have seen that bad title show up. And with a 500 http status, they would have sent accelerated requests for a while, trying to catch your server's healthy reappearance.
| 11:26 am on Mar 14, 2010 (gmt 0)|
Thanks for all your help. I have now set up a php error page that serves the 500 status.
I think what also could have happened is that the server that contains the database with my custom pages (including homepage) could have been down, while the server with the domain was already (or still) working - which is where the custom error pages are located (if that makes sense?).
Fingers crossed this will be sorted soon and not happen again!