There is no server-side solution to the problem of a missing referer. The server can only work with the information the User Agent sends it.
In any case, you can't rely on referer alone for security; it's vulnerable in both directions. One problem is the one you have already met. A browser may not send a referer, for any number of reasons. Could be browser default, user's privacy settings, even proxy or host decisions that are out of the user's power to change. Conversely, it is the easiest thing in the world to send a fake referer. Even the most dim-witted Ukrainian robot knows how to do it.
For most purposes, cookies are the way to deal with access-control issues. If it's something more serious, you are in htpasswd territory.
You also need to fine-tune your rule. It's extremely unlikely that you would want exactly the same restrictions on page requests and on non-page requests. If you absolutely must use mod_setenvif-- which I have to say would not be my choice here-- at least separate page from non-page. This is most easily done by looking at Request_URI with closing anchor.