why does it have 404's?

[HughMungus]

Or has this been answered already?

[digitalghost]

With over 3 billion pages indexed, and all of them at the mercy of of myriad technical issues, human whim and misadventure, it is simply impossible to keep from returning a few 404s.

[hakre]

on the fly outfiltering serps. google only had to request the header for each entry (maybe cached, too) in the results. so it's not belonging to an index prob... ;)

[chiyo]

>>If Google is so good, why does it have 404's?<<

1.Google indexes a page
2.The day after the site owner removes it

[HayMeadows]

The beauty of Google is that if you get a 404 error, which are ever so prominent on any given day, you can always check the cache. I find myself doing this a lot. Very rarely is it a site that absolutely never exists - Google (usually) updates often enough this does not happen.

[chiyo]

Looks like your answering your own question?

>>on the fly outfiltering serps

interesting...

Would this require qurying every reult in each of the 150,000,000 searches a day? - 1.5 Billion requests daily if there are 10 results for each query? Would this be cost efficient? Or am i missing something?

I rarely get 404 pages. I would guess that a normal person searching would only look at the first few SERPS, and seeing PR plays a good part in their positioning, it is less likely that webmasters will remove them in a hurry.

Are you getting the 404's using google as a normal user, or someone doing research on your competitive positining, entailing lots of SERPS and refinements that a normal user would not request?

[grifter]

Or has this been answered already?

Sure, in the mid 20th century, when the phone company invented "we are sorry, the number you dialed has been disconnected or is no longer in service."

I'd reserve my incredulity for 404s on Blogger ;)

[hakre]

eh chiyo, you're fully in ;). but for the google fanatics, google should add this top feature. user who don't like 404 can subcribe and enable this service in their preferences for example.

on the other hand: why pay?! this could on the fly filter out missing pages (with redundant checks if a site has a prob) and a bit of these request could be cached internally. so this won't bother googles bandwidth that much, because they have to check for 404 errors anyway.

each page with a three times 404 within a week will be put on a blacklist and filtered out on the fly without a new request. after the end of a 3 weeks period, these can be finally re-checked one-time again and then put into the 'delete from index'-list if the url still returns 404.

unfortunately that will decrease the amount of indexed pages in a very high number, so they tend not to implement it - because of bad pr (public relation) and their so hyped pr-system (pagerank) will break down - to many missing sites to re-calculate.

[edited by: hakre at 4:54 am (utc) on Mar. 4, 2003]

[chiyo]

>>I'd reserve my incredulity for 404s on Blogger <<

or Blogger pages with one entry that reads "Testing my new blog", (Posted 11th September 1999 by catlover).