A bunch of us collected our load times [
webmasterworld.com...] and came up with a distribution: [
i.imgur.com...]
Google has stated that they won't penalize unless your site is in the bottom few percent. So if you are under 10 seconds are so, you probably won't get a specific algorithmic penalty.
Rankings also appear to go down if the site is too slow for users to find it usable. The conventional wisdom here is about 3 seconds. If you can get your page load time under 3 seconds, few users will turn back when clicking your site from the SERPs, and your rankings will improve as a result.
Keep in mind, users may find your site USABLE well before it officially loads completely. Take the case of a video site where it takes 30 seconds to buffer the entire 2 minute video, but a user can start watching it even while it is buffering.
Your official load time also may be slowed by 3rd party elements that don't hinder the user at all. I've seen cases where tracking pixels and analytics scripts at the bottom of the page take 10 seconds and the user is none the wiser.
Keep in mind that there are many reasons that your site may be slow to load. Before assuming that a CDN will solve your problem, use a browser plugin such as Google Site Speed or Yahoo's Yslow to get a full list of suggestions. Many sites can be improved by using sprites to reduce the number of loaded images, by causing javascript to delay loading or load asynchronously, or by hosting images on domain or subdomain that prevents gobs of useless cookies from being sent when they are fetched.