Filipe - 1:01 am on Apr 5, 2012 (gmt 0)

Is there a preferred method for getting truly huge sites, in terms of page volume, fully indexed and up-to-date? How do the big boys do it (IMDB, Facebook, Reddit, StackOverflow, et al)? Are we presuming some sort of relationship with Google? some higher-than-10-per-second crawl rate special? Or some other technique?

One thing I considered with, say, StackOverflow is that the StackExchange network has many different sites running essentially the same product - and they just wisely broke up the site among different URLs that Google might crawl as though they were independent and at their own respective rates.

Any thoughts our resources?