If you've been following Google's arc on this, last year they did a lot of user testing around the speed of their own pages. They mentioned in a blog article that they were surprised that even fractions of a second were making a statistically significant difference in user abandonment rates.
Essentially they would like the Internet to run like a local application! Speeding up the web is a major and long-term goal for Google. Even though there may be a self-interest factor, there is definitely a strong and altruistic component, geek-style. They really want to world to have a faster Internet.
My take is that, at least in the beginning, the algo may reward fast sites, rather than penalizing unusually slow sites. They already have a pile of server response data from a decade plus of spidering. They know how fast the server side is already, and it's probably not all that good. Some of the AJAX bloat out there is becoming horrendous - and that's especially on tecchie sites!