Robots don't send referers. That includes legitimate search engines. Obviously this is not a problem-- it may even be an advantage-- if you don't want your images crawled. But as wilderness said above, there are also perfectly legitimate humans who don't send referers. And you can't identify them with some constant variable oops, uhm, well, you know what I mean like UA or IP.
The core problem is that your server can't tell the difference between a hotlink and a direct request. The referer is identical either way: It's the name of the requesting page.