I just thought I'd share my recent experience with you about page speed.
I've been doing quite well in the SERPS for ages now, but at the beginning of August i took a big hit and later discovered that my pages had slowed down. I seems that I was starting to hit the CPU usage limits on my shared server and my host throttled me back quite a bit. My page speed times doubled in WMT (up to 7 seconds). Straight away i took a big hit in traffic, which i suppose could partly be down to less repeat page views, but of course my bounce rate went up to.
the thing that surprised me was that i always thought page speed and bounce rate/page views were minor parts of the algo. i didnt think you could get clobbered on them alone, but it seems you can.
i've since moved to a dedicated server and my page speed is down to 2 secs, but i'm yet to recover after 3 weeks.