Why does the 'Google Lag' exist?

Trying to understand its purpose.

[bakedjake]

I had some in-depth discussion this weekend with some friends about the sandbox. Every theory on how to beat it kept coming back to one central problem - no one is sure why it exists.

I feel very strongly that until we have a good grasp on why it exists, it will be very hard to beat.

I don't buy the explanation that it's intended to be a method of stopping spam. Why? One, there's too much collateral damage it is doing. Two, if you accept the 80/20 principle (20% of spammers are doing 80% of the spamming), and you realize that there are multiple ways already of beating the sandbox that all of those spammers are aware of, it doesn't make sense anymore.

So, why does the sandbox exist?

The most obvious effect of the sandbox is that it prevents new domains (not pages) from ranking for any relatively competitive term. So, start thinking like a search engine - what would be the benefit of this?

[Bluepixel]

It's to stop people to create stupid websites just to earn money.

It's simple, the one's who don't create websites for money will still create websites, because they don't create their website to get google traffic. You, the spammers, won't. (I consider every SEO technique as spamming, you should leave your site as it is, and not do any work on it.)

[trillianjedi]

So, start thinking like a search engine - what would be the benefit of this?

1. Spam

Well, you've ruled this one out from your POV Jake, but I'm not sure I agree with your reasoning that there's too much "collateral damage". There may be collateral damage, but there was during Florida also. I don't think the SE's get involved in the smaller picture, they look at the big picture which is Joe Average Surfer's user-experience.

Thinking like an SE, I'd live with a lot of collateral damage if it increased the quality of my brand at the expense of the loss of sites. I'm not citing Florida as an "increase in the quality of a brand" by the way. But I'm sure that's what it was meant to be.

2. Quality of Results

This is not the same as spam. If you take a hypothetical scenario where spam didn't exist, the ranking of sites is still very important to the user-experience and perceived quality.

3. IPO

If I were a SE in this situation, I would be playing ultra-safe until I'd sold all my shares. If that meant the freshness of my results were below par, I'd settle for it over a bunch of spam and low quality SERPS.

This would be for a limited period, then I'd commence what would probably become known as the "big post-IPO update". That's where I'd throw all my new technology and recommence a program of continued development rather than "lie low till the shares are sold".

I think the reason why is a combination of the three above. At least, I can't think of anything else.

TJ

[Leosghost]

Reason =PFI,Adwords...what it definately isn't ...is lack of storage or processing ability to keep the index going or growing.
Also as greywolf speculated "enforced stablilty around the IPO"

[Rosalind]

I've always assumed that the sandbox was a symptom of spam rather than a deliberate algorithmic choice. Here's my theory:

Sites will not be put in the full index without sufficient quality inbounds. The index has grown, so perhaps the number of these inbound links required has grown. A lot of scraper sites provide inbound links, plentiful yet individually insignificant. It takes a while for these scraper sites to pick up on a new site. New pages will have the benefit of the rest of the site's anchor text, plus any scraper site anchor text linking in, whilst new domains will have neither until they get into the full index. That's your sandbox.

[DaveN]

Jake gut feeling alone here, But they are buying time, i think that pagerank is almost dead in the water, and they are crawling and re building the **NEW** NON PR Index in a new DC, if all Googles efforts are pushed into keeping up with Yahoo but without the problems that Y! have..

Then how much time would you spend on re-indexing a DB which will soon be obsolete, take florida jeeze then we had Update Rollback (sorry GG) but in eyes it was. this takes time and effort, where are aways around the sandbox and not just for the spammers we got a client site indexed in 72 hours and ranking and it's still ranking today which is well past the sandbox kill time.

What I have seen is that G is indexing new sites and is ranking them but if they don't fit their "let me in" criteria then the sites Die from the old index but every so often they run tests on the new DC with you site passes the new DB criteria then you live agian in the OLD google, problem for the masses is that they are not testing enough, if you can't find a hole and go buy a shovel and dig one!

DaveN

[Rosalind]

"Most webmasters don't have a commercial interest. They do it for fun and to help other people, not for money."
This is the one of the most infantile and silly posts I've ever seen.

Webmasters get into sites because it IS fun. And challenging. But the truth is, to develop a superior info site, one that, ultimately reaches many people and serves a ton of useful information------YOU HAVE TO BE SEEN IN SERPS. And that involves hard work aka seo. There's nothing wrong with it.

Bluepixel is right, most webmasters don't do it for money. The ones that you find at the top of the SERPS are a tiny fraction of the whole and do not represent typical webmasters at all. I imagine that those in charge of .edu and .gov domains will be less concerned with search engines, and many amateurs and hobbyists won't care either. Print, newletters, word of mouth, and radio are all ways to let people know about a website and they can work very well. At WebmasterWorld we tend to forget that you can get information at other places than the screen. I'm not just thinking of the blog that someone writes and tells all their friends about, but also the school website that gets mentioned in a newsletter, or the hobby site that is announced at a club meeting, for example.

[chrisnrae]

DaveN - very interesting theory. I think most have realized PR hasn't been "true" in a long time. You're post made me remember a line from the Boston pubcon where the Teoma rep (I believe, going from memory) responded to a comment asking if they would be coming out with their own pagerank by saying something along the lines of why would they want to duplicate an "old" piece of technology (again, totally from memory).

You're theory is the best I've seen to date. If I'm interepeting it correctly, G is making a totally new base to the algo - why spend time and effort "regulating" the old index in the meantime. But, they can't not do anything to the old index or it will turn into total crap.

So, just tweak the old index, prevent new domains from entering while still allowing new pages from established sites in and you've achieved an unnatural "stability" to the results. Webmasters are busy chasing PR and trying to figure out the ways around the sandbox - leaving them the time to focus on the new algo/index on a different datacenter - because they certainly don't want a repeat of Florida.

Nice food for thought over my morning soda. Thanks Dave ;).

[mfishy]

I am no longer convinced that the sandbox effect exists to prevent "spam" either Jake. If that were the case, essentially, they are simply providing users with old "spam" instead of new "spam".

Do those of you who believe spam prevention is the goal here honestly think webmasters have stopped creating sites because of the sandbox? Is this really a long term solution? Wouldn't an intelligent scoring algo solve all of these problems anyhow - or is this simply part of it all now? Most importantly, does anyone here honestly believe the SERPS are somehow better today than they were 6 months ago?

[trillianjedi]

Most importantly, does anyone here honestly believe the SERPS are somehow better today than they were 6 months ago?

Not for SERPS I watch, no.

But they are no worse, either.

If that were the case, essentially, they are simply providing users with old "spam" instead of new "spam".

Hmmm.... yes, that's a good point.

TJ

[hooloovoo22]

I like DaveN's theory, but have a couple questions.

1. why would they spend time completely redesigning their algo and recreating the index when they already have arguably the best search engine already? That's a HUGE risk. True, you should never rest on your laurels...but it wasn't broken and fleshing out mail, news, portal, browser, etc would seem to expand their search empire more effectively.

2. it's not as easy to become a mainstream search engine anymore, somebody with a nifty little technology isn't going to slip by the big guys so easily like in the past. If they see something that does it better and is gaining market share, pull a MS and buy it?