Why does the 'Google Lag' exist?

Trying to understand its purpose.

[bakedjake]

I had some in-depth discussion this weekend with some friends about the sandbox. Every theory on how to beat it kept coming back to one central problem - no one is sure why it exists.

I feel very strongly that until we have a good grasp on why it exists, it will be very hard to beat.

I don't buy the explanation that it's intended to be a method of stopping spam. Why? One, there's too much collateral damage it is doing. Two, if you accept the 80/20 principle (20% of spammers are doing 80% of the spamming), and you realize that there are multiple ways already of beating the sandbox that all of those spammers are aware of, it doesn't make sense anymore.

So, why does the sandbox exist?

The most obvious effect of the sandbox is that it prevents new domains (not pages) from ranking for any relatively competitive term. So, start thinking like a search engine - what would be the benefit of this?

[graywolf]

Look, after about 350 posts I don't think we are really any closer to determining why the 'Google Lag' exists

I felt the same as you back around 200 or so, but this bit in the last few days with re5earcher has rather interesting. I still believe the IPO played a role, just not a much as I previously thought.

[isitreal]

<<< I don't think we are really any closer to determining why the 'Google Lag' exists

I feel like it's closer, can't speak for anyone else, other elements too have been educational.

[randle]

I agree, there has been a lot of talk and good ideas thrown out about something that is clearly very significant. I can say I have benefited quite a bit by the thread.

Although all my new sites are still at the beach.

[drbrain]

plumsauce wrote:

The 32 bit limit is immutable in the hardware.

Sorry, 32 bits is the size of the address bus for Pentium and older ia32 chips, and only limits the amount of memory addressable. Pentium Pro and newer chips have 36-bit address busses supporting up to 64GB of addressable physical memory.

The data bus reached 64 bits with the Pentium processor:

The main registers are still 32 bits, but internal data paths of 128 and 256-bits have been added to speed internal data transfers, and the burstable external data bus has been increased to 64 bits.

The Pentium Pro is able to pull 64-bit chunks of data straight off of its caches:

The power of the Pentium Pro processor is further enhanced by its caches: [...] a 256KByte L2 cache that's in the same package as, and closely coupled to, the CPU, using a dedicated 64-bit ("backside") full clock speed bus.

All of Google's ia32 equipment can handily shuffle about 64-bit values with no problem. By now they're probably retiring the last of their Pentium III chips. (Note that the MMX instructions use 64-bit registers.)

You'll also want to check out UFS2, which is a 64-bit file system used on FreeBSD, OpenBSD, NetBSD, and other BSD derivative operating systems. The 64-bitness of the filesystem has no noticable impact on speed.

UFS2 was developed and polished in about the same amount of time since I first heard of the '32-bit problem' theory. Since UFS2 was devolped in part by Marshall Kirk McKusick, the author of UFS and FFS (a really smart guy who knows his filesystems), and GFS was developed by a team of equally (or maybe even not-quite-as) smart guys, I doubt that this is a serious constraint for Google. In fact, I doubt that 32-bitness was ever a problem in GFS, since they knew it was going to be big all along.

PS: UFS2 probably had larger problems with UFS filesystem compatibility than expansion to 64 bits.

PPS: You don't need to jump to 64 bits from 32 bits. 36 bits is probably more than enough, and gives you a large on-disk savings.