I think clickthrough is a big factor. Well, maybe not big like links, but I really think it has an influence more than is discussed on this board.
I just recently launched a site and the logs report quite a bit of clicks for a certain keyword on Google. In one day I jumped three spots on the SERPS and I don't see any other reason for the sudden jump other than lots of people clicking on it (i.e. no new links, optimizations, noticable updates, etc). The page is very new too, only about a week old, so it first appeared at spot #7 then moved to #4 after I started getting lots of clicks. From looking at the SERPS it really is one of the most relevant results for the search term, so suspect the clicks are telling the same thing to Google.