pageoneresults - 6:50 pm on Dec 7, 2010 (gmt 0)
p1r, sounds like you've never tried to move a billion documents before.
Not even close. The most I can claim is approximately 45 million documents. It took a year of deep diving through 6 layers of programming logic.
Trust me, there is a big difference.
The only difference is the number of documents and the time involved to do what it is you are doing. Moving a billion documents would be a monumental task. Its the time factor that gets extended.
I would guess there are about a handful of folks around this neck of the woods who can lay claim to managing 1,000,000,000+ documents. That's a mouthful. :)
How about checking for link-rot on 1,000 pages vs. 1,000,000,000 pages? Most methods which work on a 1,000 page site would take a full year on a billion page site.
I would think all of that is dynamic and pretty much real time. It's not something you're going to do in large sweeping tasks, it is done proactively and on a regular basis.
Analytics only allows ads on 20,000 pages to be tracked. That's only 0.002% of a billion page site.
Ah, we were talking structure and taxonomy, analytics is another topic in itself.
My guess is that most mega sites (like mine) are comprised mostly of user generated content. There are a whole host of issues that this introduces which can easily be monitored on a 1,000 page site, and not so easily on a million page site, much less a billion page site.
I guess it all comes down to the system and the personnel who manage it. I fully understand the scalability of man/woman power in this instance. UGC is another factor that in itself requires a bit of human intervention to manage. Its all scalable, you just add zeros to the numbers. :)
From the original article...
With mega SEO it's about making small strides over time that [when] grouped together they have a really big impact.
If you're managing a billion documents, you are of course taking those small strides. In all the years I've been reading here at WebmasterWorld (since 1999), I don't think we've discussed the moving of 1,000,000,000 documents. Maybe millions, but not a billion. That's a very large number. :)