I think that if there was a reliable way for search engines to track user behavior inside a website, the user behavior would be a much more accurate indication of the relevance of the website to the searched keyword. Therefore, it should be factored in for the ranking of the website for that keyword.
There are also other useful things regarding user behavior to track, such as links clicked, pages/visit...Most of the stuff you find in Analytics...
@Quadrille Why do you say "penalize"? You said that the SE misdirected you. You arrived at the website by mistake, and the website itself shouldn't care that you arrived to it because you're not their target audience. It shouldn't matter to the website if doesn't show up for that irrelevant keyword.
@dickbaker There are still ways to study your behavior when you immediately bounce off of a website. For example, did you keep going through the other websites on the SERP, did you try another similar search keywords, or did you end that particular session and went on to do something else? Having the answer to these questions could tell the search engine how relevant the information it displayed is.
At the moment bounce rate doesn't affect SE results, its only being used for Adwords. But just in case this also applies to SE's then it would be hard for every site owner, since most users they just open many websites and close them in a few seconds.
And many sites though relevant to the topic would loose if they are not able to make the user stay longer than those 3-4 secs that they are on their site.