lucy24 - 3:04 pm on Apr 5, 2012 (gmt 0)
I got the impression the OP was asking how they do it physically.
:: business with calculator ::
1 page/second = 86400 pages a day. So if you've got millions of pages, do you let them crawl like mad, as fast as they like, or resign yourself to some pages being several weeks out of date?