Googlebot, Googlebot, where fore art thou?

[ownerrim]

Just changed hosting on a site about five days ago and it seems the new dns information has propagated out. So, far slurp and msnbot have showed up, but nothing from google yet. Anyone have any idea of how long this may take?

[otech]

If you only changed IP and didnt change DNS, then I wouldnt worry too much, I have done this a few times now and it never makes too much difference to my positsions/pages in the serps..

If you changed domains too though, you should get some new back links to the new one and 301 all your old pages to the new domain..

maybe you should get a couple new links anyway... ;)

[percentages]

Is this proof that Slurp and MSN are faster crawlers?

I doubt that! In my experience Google is sometimes the fastest!

This process is about the law of averages. You got lucky with Slurp and MSN! Other times Google will get there first.

At the end of the day, look at the positive, what used to take months, now takes days, and that is good for everyone :)

[harbs]

Anyone have any idea of how long this may take?

I just did a hosting change (IP & DNS) a couple of days ago & the transition by most of the major spiders, Googlebot especially, was almost instantaneous. They hardly skipped a beat.

Only Lycos spider & almaden.ibm spider are still crawling the old site. I wonder why...

[BillyS]

I switched hosts last month. Google, MSN and Yahoo picked up the switch almost instantly. Trailers - even after several days - were those robots gathering RSS feeds.

I left my old site intact for nearly two weeks. That was the end of my contract with my old host. I timed the switch that way.

[ownerrim]

"If you only changed IP and didnt change DNS"

Not sure what you mean by this. I changed to a new hosting service which meant a new IP, and, of course, I updated my dns information with the registrar (and, as always, left the site up at the old host to be safe).

I didn't change the domain name if that's what you mean.

[Wizard]

If you didn't change the domain name, there is no reason for Googlebot to any delay in crawling. Was it visiting your site every single day and now it's gone? And you have other traffic showing in the log on new server?

[ownerrim]

Not sure if it was visiting every day, but hasn't visited since the switch to the new host. Like I said, msn and yahoo found their way to the new host. So far, googlebot is missing in action. However, I am also noticing some weird things with cache dates on other sites, so maybe it has something to do with this update. Not much I can do but wait.

[abates]

If you use google's translation tool to translate your site, does it work? (That said, I'm not sure if that tool uses the same DNS cache as Googlebot...)

[otech]

yeah, sorry, I meant Domain not DNS ;) doh!

Of course you would have to change DNS, but if you didnt change the domain name then it should'nt matter.

One thing I have noticed with some crawlers from some SE's, is that they seem to use the servers IP to determine when to crawl the sites hosted on it to some degree - I have several sites on a couple of IPs, and google bot often (not always) seems to run at about 1 crawler per IP at a time (not per domain) - as if they use the IP to determine sometimes when to crawl the domains hosted upon it...

This may just be a simple way to avoid overloading servers - but it may also be more involved than that and perhaps there is a crawl list for each IP, moreso then each Domain on the ip.

of course this is just speculating on info from my logs

[vincevincevince]

Am seeing some serious delays in Google indexing new content for new sites :S

[Wizard]

but it may also be more involved than that and perhaps there is a crawl list for each IP, moreso then each Domain on the ip

This would definitely be a reason if it were truth. I think it would be good to ask GoogleGuy about it - it's not a SEO secret but just a technical matter.

They could remember server IPs in order to reduce DNS traffic in Googlebot crawls. Then it would result in what you observe. But it's strange that it takes so much time anyway. I recently moved a site with changing both hosting and domain, and it took about a week for Google to update the index. The new domain has PR6 backlink and plenty of PR2-4 backlinks, and I used 301 with temporary PR3 backlinks to old domain.