Doc2626 - 3:28 pm on Nov 8, 2010 (gmt 0)
I think Tedster's response is spot on! As he said, Matt has stated that pageload speed is a minor ranking factor, far inferior to content. Your visitors are your major concern, IMO.
Few visitors will discern the difference between 2 and 2.5 seconds loadtime. If you're much higher than that, however, I'd definitely try to improve it.
I recently reduced my loadtime by over 86%, just by combining my js and css calls.