We've looked into this recently and our research led us to the conclusion that we can't control this; it's a client configuration issue.
i.e. if, under Windows Explorer > Tools > Folder Options, the file type is marked as 'Browse In Same Window' then it's going to do just that. We further found myriad reasons that different file-types behave the way they do... it's a nightmare!
We just gave up in the end and let the client do what ever it was going to do...
Maybe your ASP technique works because it changes the HTTP headers? For instance, instead of the web server delivering the movies as the default MIME type for the extension (say "video/x-msvideo") it tells the browser it is instead delivering a DivX video ("video/x-application-divx"?) or something unknown ("application/octet-stream"). That's purely a guess; if you have access to your server's configuration perhaps you could try "dumbing it down" by fiddling with the MIME.
[edited by: choster at 2:39 pm (utc) on June 17, 2003]