Forum Moderators: Robert Charlton & goodroi
I don't think, the "dropping sites fast" thing is a technical or "run out of disk space" error.
Remember that we all saw a BigDaddy Index with million of pages indexed from our Websites. This Index ran on the Infrastructure Jan/Feb 2006. Why should the googlers shut down so many mchines, that their core-business "searching and indexing websites" go down to 10% of its power/capability?
So i think they did a major bug or a huge problem merging some databases (the old one aka "Supplemental #*$! 2004" and the new one aka "BigDaddy Data").
Another idea could be, that they show us a MiniGoogle (1-10% of old Indexdata) to have enough idleness and Serverpower for preparing the Mega-Super-Spam-Free BigDaddy-Index in the backround.
Conclusion: Sit down and wait and dont waste time with DC-Watchin'!
Grettings from sunny Germany,
Bonneville
Another idea could be, that they show us a MiniGoogle (1-10% of old Indexdata) to have enough idleness and Serverpower for preparing the Mega-Super-Spam-Free BigDaddy-Index in the backround.
One thing to bear in mind is that Bigdaddy will have different crawl priorities. That can account for some of it.
So I would assume from this statement, what we see now is BD, as the statement admits a connection between BD and hurt sites.
I must be in a minority here. I have several sites that I'm seeing them increase greatly in the number of pages G is indexing. Recently one went from 300-650 and another from 2-350. All of this in the last month.
However my new sites seem to be stuck in sandbox land after 4 months.
MC has just posted a comment in his snuffle snuffle blog
[mattcutts.com...]
Looks like those emails we all sent GoogleGuy are getting some serious considerations.
----
last week when I checked there was a double-digit number of reports to the email address that GoogleGuy gave (bostonpubcon2006 [at] gmail.com with the subject line of “crawlpages”).
I asked someone to read through them in more detail and we looked at a few together. I feel comfortable saying that participation in Sitemaps is not causing this at all. One factor I saw was that several sites had a spam penalty and should consider doing a reinclusion request (I might do it through the webmaster console) but even that wasn’t a majority. There were a smattering of other reasons (one site appears to have changed its link structure to use more JavaScript), but I didn’t notice any definitive cause so far.
There will be cases where Bigdaddy has different crawl priorities, so that could partly account for things. But I was in a meeting on Wednesday with crawl/index folks, and I mentioned people giving us feedback about this. I pointed them to a file with domains that people had mentioned, and pointed them to the gmail account so that they could read the feedback in more detail.
So my (shorter) answer would be that if you’re in a potentially spammy area, you might consider doing a reinclusion request–that won’t hurt. In the mean time, I am asking someone to go through all the emails and check domains out. That person might be able to reply to all emails or just a sampling, but they are doing some replies, not only reading the feedback.
-----
youfoundjake, are these pages coming back all saying supplemental results? The only time I see pages come back are when they throw the missing ones into supp index for a day or so then they drop again.
With that being said would it not make more sense for them to have a "call to action" blog or even a guest post explaining what they are doing as it seems as the past several months go by they are being mentioned by Matt more and more. It seems unfair to keep on after Matt on index/crawl issues when he is the webspam guy.
Again, I really appreciate everything mat has done, but I would like to hear more from the index/crawl team in general as I think it would answer or address a lot of the concerns some may have.
A page that was 6 in the serps for a two keyword search (not big money) vanished a few days back. We had added some fresh content to the page so i originally assumed that perhaps it had hit a new sandbox filter or something and it would show position 200+ or something - anyway, it turns out that the page is gone, not showing on the google site: command and not showing if you search for the url string, yet google bot did visit the page recently according to my logs.
Same situation on a number of other pages on various other sites we work on.
In conclusion, Google does have a problem here and whilst often they like to look like they are having issues when they are not, this time i think they have a genuine data holding problem and it is affecting the serps
Amazing.
[edited by: trillianjedi at 1:59 pm (utc) on May 6, 2006]
[edit reason] Examplifying.... [/edit]
Is this the supplemental cleanup that we have been waiting for, or just some glitch?
But, then, based on my engineering studies I did some preliminary considerations. What is it that google follows:
1. having the biggest index of the world and having all information stored and adjusted.
2. beeing the first in internet search and stay first...
3. doing relevant internet search
All this points by doing their guiding principle "DON´T BE EVIL" gets me to the conclusion that they must have a problem to be solved with or without us webmasters.
So: IT MUST BE A BUG
Greets LG
Results in most of the Google DCs are the original BigDaddy SERPs (two slightly different versions of that out there), and in other DCs they are the "experimental" results that started out at 72.14.207.99 and have now spread to many other datacentres (but still very much in the minority).
There are major differences between these two sets of results, especially in the handling of Supplemental Results from before about 2005 June. In the "experimental" DCs most of those Supplemental Results have disappeared today.
Things are really looking good. What's going to happen to this forum if Google fixes its ship? No one will having anything to complain about but their rightful ranking.
It is a simple economic fact that an economy based on an instable system is a fragile economy. Since Google is the main traffic distributor, the constant instabilities in their SERPS are bad for anyone doing business on the NET.
Imagine a government would constantly rebuild the roads, no one of sound mind would accept it.
Imagine a government would constantly rebuild the roads, no one of sound mind would accept it.
I don't think anyone would benefit from that. Not Google. Not consumers. Not webmasters. No one.
So you suggest that Google sits tight and does nothing?
No the obvious conclusion is that they release to a controlling body, what they do, when and how long it takes.
Soon they will be forced to anyway, see the child porn thread etc.
Is this the supplemental cleanup that we have been waiting for, or just some glitch?
I say it is a cleanup. I am hoping all datacenters are cleaned within the next few days. This is much overdue and much welcomed to see.
And people keep tweaking sites, which then move up or down in the results. So long as the net keeps churning, it's not reasonable to blame index-churn on the indexer. So long as people keep building new roads, the map is going to have to keep changing.
But it's worse than that. A lot of webmasters aren't building "roads" for surfers, they're digging trenches, spreading barbed wire, and trying to restrict access that doesn't go through their checkpoints. It's a war zone, not a subdivision! Google is trying to map the remaining open roads, to enable people to travel past the ambush sites to their real destination.
And yeah, the maps change REAL fast in the middle of a war. They have to. And in any case, if they didn't change, your sites would all STILL be falling out of the SERPS -- pushed out by new legions of doorway spammers, new scrapers, new links on other people's sites. Your checkpoints, in other words, being shut off by people laying barbed wire to funnel the sheep through their own checkpoints.
This is the new normalcy: get used to it.
However, Google is only indexing 24 pages of our entire site, the rest is not coming up in G's DC.
Is anyone else having the same problem, my question is do I need to do something about it, or just wait for Google to overcome this update, bug whatever you call it?
maps change REAL fast in the middle of a war.
Once the war against spam mail has been won, I start believing G/Y/MS can win this war.
It's the new normalcy, get used to it
Well one should better tell G/Y/MS etc that .. :)
