Google announces Caffeine is completed

[travelin cat]

Today, we're announcing the completion of a new web indexing system called Caffeine. Caffeine provides 50 percent fresher results for web searches than our last index, and it's the largest collection of web content we've offered. Whether it's a news story, a blog or a forum post, you can now find links to relevant content much sooner after it is published than was possible ever before.

[googleblog.blogspot.com...]

[Sgt_Kickaxe]

50 percent fresher results

Am I to infer that this is primarily long tail results and by fresher Google means more recent? If so that's a spammers paradise, content like guides and how to-s don't need to be re-written but are often scraped and regurgitated and worse - mashed up.

I've personally watched a sub domain abusing MEGA SITE and 100% scraped content MEGA SITE have their traffic jump in the order of 400%. The mashup site ALWAYS has 100% fresh content because it steals it on the fly. Is this what Google feels the internet needs now?

Hope not, I can set up software to copy my own sites if the latest version gets the visit. I'm highly unimpressed by these changes.

I suppose that now we know - WebmasterWorld and similar top quality sites lost traffic because, well, Google thinks they are stale?

edit: In 1 query I was just able to find a mashup site with over 1.9 million indexed pages, all 100% copied from everywhere else online including hotlinked images. Good luck with that Google, this is deja vu a la 2003.

[scottsonline]

If google has greatly expanded their capabilities how come thousands of well respected sites are losing pages by the boatload in the index?

[TheMadScientist]

If so that's a spammers paradise, content like guides and how to-s don't need to be re-written but are often scraped and regurgitated and worse - mashed up.

This is where QDF (query determines freshness) comes in to play, and my guess is the blogspot is as much PR as anything... I think one of the things we should keep in mind is Google is probably working with a 5+ year plan, not a 'right now', short-sighted, 'but it's not giving perfect results today' idea.

My guess is they will need to make some adjustments to what they are doing and ranking with the new pages available to be scored, and will probably do so as they have in the past. Even Florida was not the end of Google, and somehow, I think neither will be the new additions to the algo, nor the faster indexing and storage system (Caffeine).

IOW: I think time plays a key role in the situation, and when they can make a '5+ year' implementation by taking what may look like a step back for a few months I think from a business perspective they would be silly not to.

[Sgt_Kickaxe]

5 years would be too long to have your content generate traffic for the sites that mashed it up, but that's just my opinion.

edit: Google claims "So why did we build a new search indexing system? Content on the web is blossoming." I agree, my content is blossoming onto a load of mashup sites (who do the mashup on the fly, instantly).

[edited by: Sgt_Kickaxe at 12:56 am (utc) on Jun 9, 2010]

[J_RaD]

yay now google only serves garbage sites and adverts at lighting speed!

If google has greatly expanded their capabilities how come thousands of well respected sites are losing pages by the boatload in the index?

is goog in self destruct mode? if the SERPS suffer people spending money on adwords will start to notice the clicks arn't worth what they are paying and will stop spending cash.

goog has forgot that all they have to do is be a good search engine and THATS IT.

[J_RaD]

5 years would be too long to have your content generate traffic for the sites that mashed it up, but that's just my opinion.

5 years is 1/2 the amount of time goog has been around, if they screw up they could be history in 2 years or less!

[TheMadScientist]

5 years would be too long to have your content generate traffic for the sites that mashed it up, but that's just my opinion.

Uh, I was talking about how they'd outgrown their old indexing and storage system and how long (at the minimum) I would guess they plan to use the new faster one. The results can be changed and adjusted and refined by the algo, but getting a more robust storage system in place was a must, and now, by changing the storage and indexing system (from Big Daddy to Caffeine) they have access to more data, which would logically and realistically probably throw a few wrenches into the results, which is something they can correct over a relatively short (months?) time, but they were basically over loading the old storage system (Big Daddy) and it needed to be changed.

Not really sure what you were talking about, because the algo and the infrastructure are separate systems... The algo basically processes and ranks the data within the storage system, and can be refined independently from the method used to store the data.

Caffeine is infrastructure, storage, organization, etc.

The algo still does the processing and ranking, so it probably needs to be adjusted to handle much more data than it had access to before.

[edited by: TheMadScientist at 1:07 am (utc) on Jun 9, 2010]

[Sgt_Kickaxe]

2 years or less? try 6 months, the spam mashup site I found made an entire copy of itself already with only one letter different in the domain! Spammers have been given a gift! buy a respected site, add a script to aggregate content from EVERYWHERE instantly, and watch your indexing jump by 30,000 pages a day.

Google just made a big mistake, as is evident by the quality sites who took a nosedive and spam sites that jumped up in rank.

edit: If I find my content ranking in the top 10 and being shown to Google search visitors via mashup sites instead of mine I would be willing to join a class-action suit against Google. I find the current mashup favoritism insulting and I wouldn't even have the ability to robots.txt block Google anymore, they would be (currently are) peddling stolen content. I still can't wrap my ahead around mashup sites being what Google wants.

[BillyS]

Well, I guess I can stop worrying about Caffeine. It's official and quite different than Mayday, which was also discussed by Google.

Thanks for the update.

[maximillianos]

Great to hear the rollout is complete. One less thing to worry about.

Good work Google folks.

[tedster]

It should now be very clear that Caffeine is the infrastructure and not the algorithm. The current algo has some challenges (putting it kindly) and its fixes will be deployed on Caffeine.

[louieramos]

Yes, Caffeine is the overall new infrastructure of Google, and the new algorithm is just a part of it.

[imbckagn]

My traffic started going back up today coincidence or just a fluke time will tell.

[TheMadScientist]

the new algorithm is just a part of it.

No it isn't.

If anyone really thinks the algo changes are part of Caffeine, or doesn't understand exactly what Caffeine is, please follow the first link in the October Update Thread [webmasterworld.com] and read through some of the articles linked beyond the first one.

The algo ranks the pages.
The infrastructure stores the information.

To say the algo changes are part of Caffeine is like saying the 'PHP' of a site is part of MySQL and because someone change to Berkley the 'PHP' is part of the new storage system... They're not... They're not the same thing.

They work together, but switching from Big Daddy to Caffeine is like switching from a MySQL to Berkley Database... The storage method and system changed, but that's totally different than the 'PHP' doing ranking of the information contained by either.

[louieramos]

@TheMadScientist

The way Google is indexing pages, filtering duplicate contents, differentiating between quality content and content farms, paid links etc affects the SERP of each sites, these are all part of the new algorithm of Google, and all these factors I have mentioned are part of Google Caffeine.

Google Caffeine is the overall program. PHP and MySQL are not the same thing nor part of each other, they are different entities. To correct your analogy "PHP + MySQL = Overall Program". PHP and MySQL can be anything from algorithm change, structure change, indexing change and whatever is involved in this whole caffeine thing.

Caffeine is a way of delivering new and more accurate search results to the public and the algorithm is a big factor in doing this.

[edited by: louieramos at 4:23 am (utc) on Jun 9, 2010]

[TheMadScientist]

@louieramos

You're not correct and that's fine...
Think as you like, but please don't mislead people here.

If you follow the link you will find the Caffeine Infrastructure change (what MC and the rest of the people at Google draw a distinction from compared to an algo change) is a change from GFS I to GFS II.

The algo is not part of the Infrastructure (Caffeine) and Mayday (the algo change) is NOT part of Caffeine... It was in place before Caffeine was. The Speed portion of the algo was not part of Caffeine... It was in place before Caffeine was. If the algo changes were part of Caffeine we would NOT have 3 threads here discussing updates and changes.

Please, don't confuse people because it's tough enough to try and give people an idea of what's going on so they can try to draw reasonable conclusions and make reasonable changes if necessary without confusing the algo changes (currently Mayday) with Caffeine (Infrastructure, GFS II).

Caffeine is a way of delivering new and more accurate search results to the public and the algorithm is a big factor in doing this.

No it is NOT! It's a new way of storing the information.

[edited by: TheMadScientist at 4:26 am (utc) on Jun 9, 2010]

[louieramos]

I am not trying to mislead people..

Can you tell me the exact definition of Google Caffeine from your perspective or by how Google defines it?

[TheMadScientist]

Read the resources provided in the thread I linked.
It goes into much more detail than I can possibly communicate in a thread.

[edited by: TheMadScientist at 4:27 am (utc) on Jun 9, 2010]

[tedster]

I'm with TheMadScientist on this one. I know there's a been lot of confusion about what Caffeine is and is not around the web. In addition to the resources about Caffeine, check out the recent Matt Cutts video [youtube.com] about the algorithmic change we call Mayday. He says it very clearly - the algorithmic change has "nothing to do with Caffeine".

[louieramos]

That's fine, I understand and I have read and seen those sources too, but based on my experience in working on large sites and competitive keywords, the whole caffeine update has affected the SERP and the way Google indexed pages, links so fort so on.. They probably not admitting it yet, but soon they will, the data they have collected for Google Caffeine changes have somehow reflected the SERP, I have seen it.

They have given a hint too.. "Matt says this is like changing the engine on a moving car. For now, it’s all about indexing. The suggestion is that later it could have impact on rankings."

[TheMadScientist]

@ tedster, thanks...

@ louieramos Sorry for being so blunt, but I'm actually trying to help people understand there is a definite distinction between the algo changes and Caffeine. Caffeine will be a storage system for years into the future, which once in place will not change, but the algo generating results stored in the Caffeine system will continue to change on a daily or so basis... The algo changes, the method of storing information should remain stable for years into the future, which is a huge difference.

They could not use the old file storage system because they were out growing it, so they changed to something much faster and more robust (Caffeine), but the algo is still the same, and will need to be adjusted, which is totally different than adjusting the storage system holding the information the algo processes.

[louieramos]

Its all good, its a great discussion..

I know it has as lot to do with storage, that they can collect data 10 times faster or so and produce more updated and realtime result but I am not completely sold to what they say because of what I have been seeing. Peace.

[TheMadScientist]

I understand your point a bit better, but IMO they are flooding an 'old' algo with information and sectioning the results, so before (Big Daddy, GFS I+) there were parts of the index updated in 'near real time', but now they have the ability to update everything in 'near real time' and it's not what the algo has been doing, so they significantly increased the information available to the algo and started running it way faster, along with a change in the way long-tail searches are ranked, so what happens?

Everything looks all 'out of whack' because of the speed and information increase (or the information for a particular sub-set of results possibly decreases, depending on the sectioning of information), which means the algo needs to be adjusted to work with the new storage system more reliably.

I think we're seeing as much 'floating spam' as we are because they haven't adjusted the algo fully to Caffeine and that's what I would expect them to be doing over the next weeks and months, so I think it's basically the opposite of what you're thinking and the issue is the algo has not been fully adapted to the Caffeine 'way of things' yet.


BTW: You're right it's been a fun discussion. Peace. :)

[edited by: TheMadScientist at 5:12 am (utc) on Jun 9, 2010]

[CainIV]

What is interesting, and others have pointed out, is that indexing has hit an all time low on some websites.

Which is somewhat of an 'about face' to what Google is purporting to be the real strengths of Caffeine.

I am however noticing an increase in the number of pages for some websites over the last three to four days. Hopefully this is part of a longer term 're-crawl' of pages which were previously dropped.

[raj1094]

we analyze the web in small portions and update our search index on a continuous basis, globally

The above point is from the article "Our new search index: Caffeine" fromin Google blog spot.

My question is due to this updates can we see regular changes in SERP or Can we come up with some new sites every time we make a search in Google.

[Bewenched]

My traffic started going back up today coincidence or just a fluke time will tell.

I've gained back about 5% more traffic today .. still down about 16% from normal, but it's at least looking hopeful. However from what I'm seeing.. links from social networks to specific pages get Google to find them much faster now. I'm certainly not condoning spamming those sites, but possibly troubled pages, or new items might benefit from this type of link.

[sid786]

I lost traffic during the recent May updates and also my main website was injected, hacked by some russian guy! The loss in traffic was around 30% for me. That was 2 months ago and there was no increase whatsoever in the traffic.

I had a gain of 15% in traffic starting yesterday -- its 20% today, and that certainly makes me happy. I guess its time to thank Google for the Caffeine implementation :)

[member22]

From what I understand the new infrastructure is in place but the algo update ( re - ranking of websites ) is not done yet, is that it ?
Does anyone know when can we expect the re- ranking of websites ?

[AnkitMaheshwari]

@member22, IMO with Caffeine, the re-ranking is always happening with 50% fresher results. (Faster indexing, faster calcuation of PR, inbound/outbound link, hence faster changes)