A site (ranked above mine for a keyword phrase of mine) isn't using any of the "obvious" spam tactics but it is utilizing some misleading ActiveX controls and attempting to force users to download an executable (Windows) file.
Is this (in a sense) spam or just a terrible idea?
If I understand you correctly, you say the tactics you find problematic, are not aimed at getting higer rankings? You just don't like what the site offers to it's users? If so, how is that anything to do with spam?
Are searchers arriving at the same URL that's appearing in the SERPs, or being redirected to a forced malicious download?
Searchers click to look at a site. Unless it's clearly indicated that it's a direct automatic download and that is exactly what searchers are looking for, it's deceptive and irrelevant. Google is a search engine, not a site providing links to downloads.