Whatever's worked into Google's algorithm regarding page load times is probably a little more complicated than an average figure such as displayed in Google Analytics. I would recommend focusing on what's acceptable to your visitors, and generally on making your site as fast as possible.
Whether an avg. page load time of 4s is acceptable depends on the content you're serving, and on things like the distance between the server and your visitors. Hopefully, Analytics will soon allow the selection of Site Speed as a secondary dimension so that we can easily see which users are experiencing longer load times, where they're located, etc. You can already see which pages take longest to load, so to improve the overall experience you might want to have a look at those first.
Having said that, the Avg. Page Load Time (sec) for the one site I'm trying this on is currently 0.6s.