"Peeking Into Google" - crawl speed limitations

[moltar]

I was just reading an article "Peeking Into Google [internetnews.com]", it's almost a year old, but one thing made my light bulb go off.

One surprising limitation is we can't crawl as fast as we would like, because [smaller] webmasters complain

I always thought that the reason they indexed really slow was that they are limited on the bandwidth, but it turns out to be the wait time intervals between fetching pages from the same sever play a huge role.