Google and DNS

[PunkJazz]

Does anyone know why google is incapable of refeshing dns info on a site for as much as 45+ days after a domain moves? Why do they continue sending their bot to an old ip address instead of the one everyone else on the internet is looking at? How can this do anything but degrade the quality of their product? I assume it must be intentional since there should be no technical challenge in keeping up with the rest of the web. In fact I find it REALLY bizarre that google has this seemingly idiotic problem. What am I missing?

[BigDave]

If you read the original papers on Google, you will find that even in the theoretical stage, where they were working on it as a school project, they basically created a DoS attack on Stanford's DNS cache when they ran their crawlers, and this was for only a few million pages.

Yeah, it would be nice of they fixed it, but it might not be quite as simple as you think.

[mcavic]

That may very well be true.

But, why not have each crawler run its own DNS, have each one query the root servers, and cache for a normal amount of time?

The load on the root servers might be higher than average, but it's their job to handle load. And the load on the web sites' DNS servers would still be lower than the load of the web servers themselves. ?

[Mohamed_E]

The real problem is that, according to the spec, the hostmaster can set the TTL (time to live) to any value. Many chose idiotically small values, and I believe that Google would be fully justified in ignoring those values.

OTOH 45 days is much too long. Upping shorter TTLs to somewhere between one and five days would strike a nice balance between reducing the load and keeping up with (admitteldy infrequent) server IP changes.

[mcavic]

Agree completely, Mohamed.