Page De-indexed then Re-indexed on a Daily Basis

[PghJoe]

I have an odd issue going on. For 16 months I have had a keyword rank on Page 1 in Google. About 1 month ago it disappeared. The page must have been de-indexed. It didn't show up in any searches I did, including looking only within the site.

I made content updates on the page, re-submited my sitemap.xml file and even got a new link. Nothing seemed to bring the page back. Suddenly, one day the page was back. Two days later it was gone again. Today it back, but yesterday, the page could not be found.

I work at a large company, and our site is crawled daily.

The only change that I can place to all of this is a server change from about a month ago. My network guys set up a load balanced server system. The site exists on two servers. A page request can come from either server regardless of the server the last page came from. Each page request comes from (basically) a random selection of the two copies.

Could this cause canonical issues? I read that Googlebot bases crawling on domain not I.P., but could separate boxes cause this issue?

This is only happening with Google. Rankings and indexing is consistent from Bing and Yahoo. No changes with them.

Any thoughts?

[tedster]

It's more likely to come from the load balancing issues that Google executes on their system. No one knows the total number of servers they use to create their rankings, but it is probably more than one million. One report from 2008 [webmasterworld.com] estimates that a single Google query "travels across 700-1000 machines"... and the copies of data that are maintained at each of their data centers are not in parallel, and not without flaws and bugs, either.

So one explanation could be that currently, the data at one or more of Google's data centers is missing your page, but others have it. Depending on how their load-balancing is executed you either do or don't see it.

How one copy of their data ended up missing the page is not easy to diagnose. but since it came back today, you have reason to hope things will stabilize in your favor.

[aristotle]

Maybe these types of problems will be corrected after Caffeine is fully implemented.

[PghJoe]

tedster - Great thought. I didn't really consider that the issue may be on the Google end, but that does explain many of the issues we have seen.

As of this morning, we are ranking about where I expect and for the correct page.

Thanks for thoughts!