Sadly, some people will always click on result #1, even if it's clearly off topic. We currently have a number 1 ranking for a way off topic term and people are clicking it.
I get that, but I wouldn't just lay that off to Google. For years I've gotten a ton of UK traffic at certain times of the year because we have a lot of cities with the same name as prominent UK cities (Birmingham, Manchester, etc) People over there type in the same search strings (cityname + keyword) and my sites come up first, and UK people, apparently missing state name all over everything including the domain, click on the result. But that's a human thing, not a Google thing, and I don't think it's necessarily a Panda thing either; I've seen it since at least 2003.
If "mismatched traffic" is your problem, you could look for ways to make it an opportunity. Eventually, I put up some pages with some UK information for my UK traffic, plus a few outbound links to get them closer to what they want. So at best, they'll click on an ad on that page, or at worst, they'll follow MY links out, and not use the back button to search in Google again.