They've moved it into AdSense also as part of the Scorecard - and with some really desperate measures (IMHO) suggested to save a few milliseconds. So it must be something they consider to be kind of important.
It's all relative, like RhinoFish said. It may hurt your rankings if the slowness really interferes with the user experience, but some sites are just inherently slower than others (e.g. informative vs. rich-media content).
Whether it affects your rankings or not, load times are always worth improving. Pay special attention to your Time to First Byte (TTFB), i.e. the time taken between connecting to your server and the first byte (of the page) received by the browser.
In Webmaster Tools, there's a graph called "Crawl Stats" that shows you how long it takes Googlebot to download a page from your website. That's a rough indication of the TTFB plus the added latency, depending on where in the world they crawl from and where your server is located. Anything above 2s I, as a user, would consider slow, but Google is no doubt more forgiving ;-)
So, it may affect your rankings if performance is really poor, but in that case you should be more worried about the user experience.