Welcome to WebmasterWorld Guest from 188.8.131.52
However, the site is all working fine and using the header check shows the following:
HTTP/1.1 200 OK
Date: Sat, 21 Jun 2008 07:43:18 GMT
Last-Modified: Sun, 04 May 2008 22:12:25 GMT
Keep-Alive: timeout=5, max=100
X-Pad: avoid browser bug
Anyone know why this could be happening?
Google having a temporary 'glitch'?
We're talking IT here, so anything's possible ;)
However it is highly unlikely in this instance - a 404 is a response from your server. Your server has supplied this information to Google, otherwise the error would be different.
Looking at the exact request by Google and your server's response in logs should show where the problem lies, in any case.
a server outage
IMO this seems like an inadequate response from a hosting company. There are very few 'server outages' that will cause a server to issue a 404 response.
Again: check your server logs, which will tell you a clearer picture of what happened to your homepage.
The first homepage 404 error occured on the 13th. Now it shows the 18th. Indexed urls dropped from roughly 24000 to a little over 4000. In my case, it hasn't effected search listings or rankings (yet), however, something is going on that needs attention.
[edited by: Key_Master at 10:25 pm (utc) on June 21, 2008]
This is apparently just a break in the 'data feed' into GWT itself, because none of the pages --404 or otherwise-- have budged in the SERPs, and no other ill effects have been noted. This is true for both the main domain (HTML), and a mobile-device-specific site on a subdomain (XHTML+XML/Mobile), so two different crawlers (Googlebot, and Googlebot-Mobile) would have to be broken if it were a true crawler issue.
My GWT report updated just a few hours ago, and it looks like ~80% of these bogus errors were cleared.
I'm basically ignoring the "Not Found" report until they fix it... But it is rather annoying.
I think it's Googlebot-Mobile that is broken. It doesn't seem to following 301 redirects. Maybe it's mistaking 301's for 404's. I think standard Googlebot is working fine, however, if it doesn't recrawl pages that Googlebot-Mobile thinks are 404's they will drop from the index.
And I don't see any 301 redirects encountered by Googlebot-Mobile on the main (non-mobile) domain either, since "wrong" links to the site are very few and far between (The main site was 301-canonicalized at birth, so it's rare to ever encounter a non-canonical link).
So... I have no data, except that both sites are showing bogus "Not Found" errors in GWT, and neither seems to have fed any Googlebot a 301 (or a 404, or a 500) in the past month. It's been all 200 and 304 responses here... Frankly, I think maybe a fail-to-connect-to-database error in the GWT 'evaluator' must be handled like a crawled 404 or something... ;)
High level pages are much less likely to be dropped from the index due to this 404 issue but deeper level pages with less page rank are being scrubbed from the index.
The main site was 301-canonicalized at birth, so it's rare to ever encounter a non-canonical link
Lucky you :) I have dmoz listings pointing to the www.example.com and the example.com domains so no such luck for me.
The site then had a 301 redirect added a few days after that error message appeared in WMT. This was to catch any traffic following the duff link and redirect it to the correct page.
A month later, and WMT continues to show that URL as a 404 Error in the Crawl Errors report, even after the page that contains the duff link on, has been crawled again.
Longer discussion is in: [webmasterworld.com...]
I had the 404 homepage error last week when this thread was started but it has since went away. Oops this thread was started last MONTH...
[edited by: BradleyT at 5:52 pm (utc) on July 21, 2008]