A bunch of us collected our load times [webmasterworld.com...] and came up with a distribution: [i.imgur.com...]
Google has stated that they won't penalize unless your site is in the bottom few percent. So if you are under 10 seconds are so, you probably won't get a specific algorithmic penalty.
Rankings also appear to go down if the site is too slow for users to find it usable. The conventional wisdom here is about 3 seconds. If you can get your page load time under 3 seconds, few users will turn back when clicking your site from the SERPs, and your rankings will improve as a result.
Keep in mind, users may find your site USABLE well before it officially loads completely. Take the case of a video site where it takes 30 seconds to buffer the entire 2 minute video, but a user can start watching it even while it is buffering.
Your official load time also may be slowed by 3rd party elements that don't hinder the user at all. I've seen cases where tracking pixels and analytics scripts at the bottom of the page take 10 seconds and the user is none the wiser.