dataguy - 4:01 pm on Dec 7, 2010 (gmt 0)
Are there really that many differences when dealing with 1,000 documents or 1,000,000,000
p1r, sounds like you've never tried to move a billion documents before. Trust me, there is a big difference. My site caching system creates a fully-functional HTML file for every page. Trying to move these files will choke a server for hours, trying to recreate them can take weeks.
How about checking for link-rot on 1,000 pages vs. 1,000,000,000 pages? Most methods which work on a 1,000 page site would take a full year on a billion page site.
Analytics only allows ads on 20,000 pages to be tracked. That's only 0.002% of a billion page site.
My guess is that most mega sites (like mine) are comprised mostly of user generated content. There are a whole host of issues that this introduces which can easily be monitored on a 1,000 page site, and not so easily on a million page site, much less a billion page site.
These are all issues which have to be taken into account on a mega site, and they are just the tip of the iceberg.