There are several ways to stop direct access, none of them trivial. You cannot use the HTTP_REFERER though, because media players typically do not send a referrer. So that variable will almost always be blank.
Here are two typical methods in outline form:
Set a cookie on the page(s) where your site links to your mp3 file(s). This cookie's meaning is "visitor has requested the mp3 from one of my 'authorized' pages." Check for that cookie before serving your mp3 files. The cookie can be set and checked using mod_rewrite or set on the "pages" themselves (if those pages are generated by a script) and checked by your script that serves the mp3 files.
Change the URL-path to the mp3 file(s) or the directory in which you keep them. You can do this manually (like once a week) by renaming the file(s) or their directory, or use a script to do it. You will also need to change the links on your pages, and reduce the caching time (or disable caching) on the pages that link to the renamed files for some period of time before making this switchover, so that the links do not appear broken if someone has a copy of the linking page in their browser cache. Be careful with this, as the page caching time must always be short, and you will lose the benefit of caching while you are waiting for your visitor's browser caches to expire before switching the URLs. The cookie method above is superior because while it may be perceived as more difficult to implement, it is much easier to administrate.