Speed of bots and site grabbers

Why so much faster than a browser?

[geekay]

How come SE robots and site grabbers/web site copiers can download big files at a rate that is so much faster than my browser and 2MB DSL achieve?

[jdMorgan]

Mainly, four factors, I'd guess:

No browser rendering engine to slow things down -- just saving the data to disk.

Multiple simultaneous requests -- your log file shows time of request only, not duration.

Machine dedicated to the job -- no other tasks running.

Some downloaders may be located on a major backbone -- Fiber makes DSL or T1 look slow.

Jim

[Matt Probert]

And many robots don't bother retrieving images. Our own spider just grabs the bare text from pages it is investigating.

Matt