|Does site load speed affect SERPs?|
IF my site is slow to load would it affect its ranking in SERPs?
Yes, but you've asked a very broad question. How slow, compared to which others, how much is affected, mobile vs desktop, etc.
So many factors determine the outcome, but yes, speed is among them. It's a question of degree.
They've moved the recommendations to increase your site speed into Google Analytics, fix the worst issues.
Where the point of diminishing returns is, is a very complex question though.
They've moved it into AdSense also as part of the Scorecard - and with some really desperate measures (IMHO) suggested to save a few milliseconds. So it must be something they consider to be kind of important.
It's all relative, like RhinoFish said. It may hurt your rankings if the slowness really interferes with the user experience, but some sites are just inherently slower than others (e.g. informative vs. rich-media content).
Whether it affects your rankings or not, load times are always worth improving. Pay special attention to your Time to First Byte (TTFB), i.e. the time taken between connecting to your server and the first byte (of the page) received by the browser.
In Webmaster Tools, there's a graph called "Crawl Stats" that shows you how long it takes Googlebot to download a page from your website. That's a rough indication of the TTFB plus the added latency, depending on where in the world they crawl from and where your server is located. Anything above 2s I, as a user, would consider slow, but Google is no doubt more forgiving ;-)
So, it may affect your rankings if performance is really poor, but in that case you should be more worried about the user experience.