Independent (not by Google) tests have shown that more than 3 seconds is a problem for users. And Google's own tests show that shaving even 0.1 second off their SERPs load time increases user engagement.
I have worked with pages from major enterprises that have horrendous load times but excellent content. They often have bounce rates in the 80% to 90% area. The first "layer" of improvement is usually not hard to achieve and I've seen going from 10 seconds to 5 seconds drop the bounce rate down from 85% to 45%, with an incredible lift in conversions from the page (in that case, lead generation).
Getting below 3 seconds is, IMO, the holy grail. As robzilla said, Google's speed metrics are pretty complex. As a direct ranking factor, load time is minor. But it's indirect effect may be huge, especially with user engagement becoming prominent these days.
The big deal for me is not what is acceptable to Google. It's what is acceptable to the visitor, what serves them best. I prefer to move mountains for visitors rather than for Google. In this case, the two are well aligned anyway.