Browsers not sending referer information to my server

anti-hotlinking htaccess code implemented, problem when referer withheld

[mobsterman]

Hi I have anti-hotlinking htaccess code implemented. Recently I've been getting lots of 403 forbidden responses. All these 403s have the common characteristic of a blank referer field.

Can someone tell me why some browsers withhold referer info? I need to explain this problem to these frustrated people who get the 403 message. Has it got anything to do with firewalls and anonymisers? What can I suggest to these people to do to avoid the 403s?

Appreciate any input on this problem.

Thanks in advance

[Slade]

Have you got legitimate users actually asking why they're getting 403's, or are you just seeing them in your logs?

Setting up anti-hotlinking code is most probably going to result in 403's. That's what it does, prevents non-your-domain referrers and blank referrers.

Are you anti-hotlinking images or content pages?

To Others, not necessarily the OP: Is there any legitimate reason to send blank referrers internally on a website? (I'm not talking about your original hit into the site, I mean while navigating the site.)

[encyclo]

Welcome to WebmasterWorld mobsterman. Some users (such as myself) choose to withhold referers for privacy reasons, but most users who block referers do so unintentionally. As you say, some firewalls and proxies do not send referers, as do some very popular ad-blocking and security programs in Windows, such as Norton Internet Security.

As such, you should not rely on referers being present for your site to function. The best way is to permit blank referers as well as referers from your site.

RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://[^.]+\.example\.com [NC]
RewriteRule \.(gif夸pe?g在mp如ng)$ - [F]

[mobsterman]

Thanks for your responses folks. We set up the anti-hotlinking for images & I was responding to legit users (or so I think).

Here's the code in the installed htaccess:

RewriteCond %{HTTP_REFERER} ^$ [OR]
RewriteCond %{HTTP_REFERER}!^http://(www\.)?mysite如hotoforum1如hotoforum2)\.com [NC]
RewriteCond %{HTTP_REFERER}!^http://(www\.)?photoforum3\.net [NC]
RewriteRule \.(jpg夙if夸s存wf)$ - [F,NC]

I found this code somewhere on this forum some time back & it's been working well. Or maybe nobody ever told me they couldn't access my image files before :)

Anyway, what I understand is that this code works by ensuring that browsers requesting these image files came in by following links from the approved URLs. And blank referer requests are direct requests (as in typing the image file's url directly into the browser) and treated as if they're not on the approved list. Is that right?

Encyclo, thanks so much for your code ... is the [^.] in your RewriteCond rule the part that allows blank referers? If I implement that, wouldn't it negate my second objective of preventing direct downloads? It appears I cant have the cake & eat it as well, huh?

[jdMorgan]

No, you can't block all hotlinkers AND have your site work for people who use security software or who access the web from behind a caching proxy (such as many corporate users, many ISP customers, and all AOL users).

The good news is that by blocking just *some* of the users loading images from your site because of a hotlink on another site, you generate complaints to that hotlinking site's Webmaster, who may then remove the hotlink.

It boils down to this question: Do you want to save *all* that hotlinked bandwidth while making your site look broken to some of your legitimate visitors, or are you satisfied preventing most hotlinking, keeping all of your legitimate visitors happy, and making the hotlinker's site look broken to many or most of his visitors?

Also, the code you found previously is "non-optimal" because the first RewriteCond is logically redundant; A blank referrer is certainly never going to match your own domain or your own www subdomain. Use this modified version of the code encyclo posted instead:

# If referrer is not blank
RewriteCond %{HTTP_REFERER} .
# and is not my domain or a subdomain of my domain
RewriteCond %{HTTP_REFERER} !^http://([^.]+\.)?example\.com [NC]
# Then return 403-forbidden for image requests
RewriteRule \.(gif夸pe?g在mp如ng)$ - [F] 

The "([^.]+\.)?" pattern, modified slightly from that posted above, is intended to allow any subdomain of your domain (e.g your www subdomain), or the domain itself (without a subdomain), as a valid referrer. This pattern makes it unnecessary to have two RewriteCond lines -- one for your "www" and one for your "non-www" hostname.

In regular-expressions language, it literally reads, "one or more characters not equal to a period, followed by a literal period, and all of the preceding is optional." This allows www.example.com, example.com, and, for example, test.example.com as valid referrers -- in other words, your domain and any subdomain of your domain will be allowed.

Jim