Respecting TTL

Shouldn't Google respect the TTL of Web Site DNs entries?

[Made In Sheffield]

I recently moved my site to a new server (no more shared hosting), I knew before I did it that Googlebot would probably take a while to get to know the IP change so this is not an "OMG Google isn't indexing me anymore" post.

The thing that occurs to me is if systems round the world started ignoring TTL and just using the same IP address for months Google would be in trouble (I believe a low TTL on www.google.com is key to their load balancing?).

Shouldn't they really do what they expect everyone else to do and respect the DNS standard by rechecking entries once they expire?

Sorry if this has been raised before. I know it's not just Google who do it.

Cheers,
Nigel

[Kimberly]

Does Google not track by Fully Qualified Domain Name (FQDN), and not IP address, so should pick up the new site automatically?

[Tropical Island]

Our experience is that they are very slow. In our case it was more than two weeks before Googlebot found us (and we missed the deep crawl. They are not alone. Fast had us indexed for 6 weeks as "this site is temporarily out of service" which was the 404 page for the old IP address. We went through a very nervous month while they all caught up. Knowing what I know now I would have left the site intact on the old server until I was sure the bots had found the new one. Just a dumb uninformed mistake.

[kwngian]

Hello

What you may be experiencing is what I am experiencing on a monthly basis - because I am on dynamic IPs.

I don't seems to face problem with google at all. They normal revisit my site within 2 days, especially since 2003. The other one I don't have problem is Inktomi - Slurp. These 2 search engines update their DNS faster than my ISP.

Fast is quite "Slow" on their DNS update. Altavista Scooter - don't even know how often they update.

kwngian

[Made In Sheffield]

But what I'm saying is they should update their DNS at the interval specified by the TTL on the DNS record and not choose their own timings to suit themselves.

Like everyone else on the web does....

Cheers,
Nigel

[Mohamed_E]

> But what I'm saying is they should update their DNS at the interval specified by the TTL on the DNS record and not choose their own timings to suit themselves.

In theory they definitely should. However, many DNS admins set inappropriately short TTLs, and Google does a lot of DNS lookups :) In the real world what they do seems inevitable, but it would be nice if they had a mechanism by which webmasters could inform them of an IP address change.

[Kimberly]

Gee, I would have expected that Google would look to your DNS server to resolve, not look to their DNS server. Not so?

[kwngian]

Any reason why you're thinking that google is not following the TTL for your zone?

In my case, it is my ISP that is not updating their DNS especially when there is no change in the zone serial no, they would assume that it is the same as before even though it has changed.

kwngian

[dwilson]

Gee, I would have expected that Google would look to your DNS server to resolve, not look to their DNS server. Not so?

Google (presumably) uses DNS just like your computer & mine do. When I go to google.com in my web browser, Windows first checks if my machine already knows Google's IP address. If not, my computer asks its DNS server -- run by my ISP. If that doesn't know, it asks a root server for the IP of a .com DNS server. It then asks the .com server for the IP of google.com. Eventually, my computer gets the answer back.

But each DNS server along the way caches the results of its lookups. It makes no sense for my DNS server to ask a root server every time I or anyone else using it hit Google.com.

Whoever controls a site's DNS entries sets a Time To Live (TTL) value that tells other DNS servers how long they are supposed to cache the answers. After that time expires, they are supposed to come ask again in case the information has changed.

Two problems exist with the system. As was mentioned earlier, some DNS servers specify a very short TTL -- say 5 minutes. It's crazy to think that the cache should expire every 5 minutes. Honoring that TTL results in extra network traffic. So some DNS servers decide not to honor the TTL. They will expire the cache when they so please. Google apparently does this as does AOL.

A better solution to the first problem is for DNS operators to set a lower minimum on the TTL's they will honor ... say, 1 day.

[Databoy]

Consider that when you go to Google's website, you do walk up the DNS heirarchy, using first cached info, then your default DNS Server, which will probably resolve Google's IP Address by making a recursive query of Root and then Google Name Servers on your behalf. You are right about all this.

But, consider that once you are at Google's web page, when you click on any website listed, you are hitting not Google's DNS server to resolve names, but rather the DNS server for the site you are accessing (assuming you haven't already got it cached).

As to whether Google is even using it's publicly listed name servers when it does it's thing, who's to say (somebody on the inside) Consider that they might not in deference to their customers. And how long they cache for--not certain. I maintain a small number of sites that we occasionally move from one server to another, and I have seen no evidence that they are exceeding my TTLs.

I have found that in a fair number of these cases, the real problem is right on your local machine... cached.

Assuming your running some flavor of Windows, this can often be overcome by typing at the command prompt:

ipconfig /flushdns

Hope this helps.

[Kimberly]

Thanks, Databoy :)

[Made In Sheffield]

The reason I'm saying they don't respect the TTL is that within 1 day (the TTL) of the IP address change the old site gets no visitors other than bots, all humans are coming to the new site IP.

Cheers,
Nigel