Is there a way to see where those visitors are really coming from?
I'm pretty sure this not possible at the moment.
However, search for your displayed URL and the first line of your Adwords copy text (in quotes) and you might come across it in SERP.
I think they've fixed it now, but some Adwords got indexed in Google last month when Googlebot went spidering partner sites.
Hope its not against TOS to post this url here.
From the publisher's side (I've been running 468s on one site since they announced the program -400k+ impressions/mo.), things are surprisingly stable. Defaults haven't really trended up all that much, which would be the case if advertisers were dropping out of the program, so my guess is that they are satisfied with the CTR --or don't know enough to be dissatisfied.