Here's how I would code it after a little bit of modification. RewriteEngine on
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^https?://(www\.)?mydomain\.com/ [NC]
RewriteRule \.wmv$ - [F]
This won't block user-agents with blank referrer headers but it will cut down on the number of unauthorized downloads from hotlinks.
Based on the number of downloads you report, it's possible that your videos are being indexed by media bots. Maybe a robots.txt exclusion is in order for your video files. Also, you should compare the server headers for the html files and the wmv files. It's possible that browsers are caching the html files but are not caching the wmv files. If so, this can be easily fixed.
With a simple modification to the above .htaccess code, you could easily create a short promotional wmv clip and redirect hotlinkers to it. This could include free advertising for your site or info to direct the visitor on where to access the videos legitimately.