tedster - 2:14 pm on Nov 6, 2010 (gmt 0)

First thing I'd say is that "penalty" is not the best way to think about page speed. It's a ranking factor along with many others. Matt Cutts made a comment in one of the Google Webmaster Help videos that it only makes a difference in something like 10% of all searches, and then it acts somethinglike a "tie-breaker" when two results are very close.

Now that certainly may change - but I've worked with clients whose sites were reported to be at least as slow as your report says, and they were doing pretty well. We fixed a lot of things at the same time as we worked on Page Speed, so I can't isolate that one factor, however.

I'd say that visitors are punishing slow sites much more than Google's algorithm ever will. So improve your speed for you visitors' sake and your business will improve. Even more, when visitors like your site, they are more likely to help you promote it on their own and your links and mentions around the web will grow, too, and that REALLY helps.