Blocking hotlinking nowadays

Doesn't work anymore like it used to

[geekay]

This is not a question, as I presume there is no answer. It is just a note. It seems to me that with the increased popularity of Firefox, stopping hotlinking with mod_rewrite is becoming less efficient. It worked so fine before.

With Firefox it is easy even for ordinary surfers to block the referrer string in their browser. But for a number of reasons no webmaster wants to refuse file requests with an empty referrer. Thus files slip off where they should not have gone. I guess this is a growing trend.

[jdMorgan]

Blocking hotlinking by HTTP-Referrer has always been problematic, since corporate and ISP caching proxies often block referers (there is no meaningful referrer for a cached page inside the corporate network that would not expose inforamation about that internal network -- a potential security risk).

Then we add Norton Internet Security and the Mozilla browser plug-ins that enable referrer blocking and user-agent spoofing, and the problem gets worse.

Why allow blank referrers? Get any traffic from AOL? Well, you won't if you block blank referrers, and many other ISPs use caching proxies/filters that won't provide a referrer as well. In addition, people serving video on their site, or using JavaScript to request image loading have another problem. Media players and JavaScript requests usually *never* provide a referrer.

Referrer-based access control has never worked all that well against determined people who wanted to steal your content. It is just an easy way to prevent casual theft of content of bandwidth. This is similar to those silly JavaScripts that disable the "Save As" browser context menu. I have to laugh when I encounter one of those, especially if the content is something I need to save and print in order to *buy* something from the site (like I have to print it out to ask my mother-in-law if that's what she was looking for).

If you want security, it's never easy. The solution to the problem is to use a script to serve your images, and only serve images if the user's browser provides a cookie that you have set on a page that is authorized to include the image. Then, set the cookie expiration for only a few seconds, and set the cache-control header for the image so that is is uncacheable. Note that this will probably result in a huge increase in your server load, since images will have to be re-requested from your server every time the page is loaded or reloaded. The script and cookie stuff adds complexity, and cache-control costs bandwidth. But even moderate security is never easy, because the Web was designed to be 'open.'

Jim

[bcolflesh]

The solution to the problem is to use a script to serve your images, and only serve images if the user's browser provides a cookie that you have set on a page that is authorized to include the image. Then, set the cookie expiration for only a few seconds, and set the cache-control header for the image so that is is uncacheable.

Can you elaborate on this - point to some example scripts, etc?

[jdMorgan]

Actually no, I don't have an example at hand. It's just a common method of using a cookie to authenticate a user. I'd imagine you could take the picture-gallery script from sourceforge and use that as an example, although that script is *way* more complicated (feature-rich) than necessary for basic image-serving.

The jist of it is:

1) Use JS or server-side code to set a cookie on an 'authorization page' or for a 'permission domain' (the page's directory and everything below that). Nothing complicated here, your script determines what this cookie means.
2) mod_rewrite all protected-image requests to a PHP or PERL script.
3) Have the script check the cookie, and if valid and if not expired, then:
4) Have the script output correct MIME-type headers, open the requested image file, and pipe the output to the browser.

As noted before, send cache-control headers with the images to mark them as non-cachable (or mark them to expire after only a few seconds) and set the cookie to expire after some reasonable amount of time. If the cookie is absent (expired), have the script reload the html page that sets the cookie and 'brands' your site (or requires a login -- depends on your needs).

mod_rewrite, mod_headers, and mod_usertrack may come in handy.

Jim

[geekay]

My own RewriteCond has always allowed blank referrals. I've never dreamed about blocking them, and after Jim's explanation I think no-one else will contemplate that either. I just noted that Firefox has now made it easy for everyone to take advantage of that "loophole" (i.e. blanks can't be blocked).

However, contrary to Jim I don't find hot-linking a security issue, but rather a bandwidth nuisance. For some reason posters to a variety of blogs and forums seem to be very fond of my images (I have a lot of them). Every time someone reads such a thread my (and others) jpeg's are requested. If they would simply copy the jpeg file, instead of hot-linking to it, I would be happily unaware of what's going on. But that would, of course, be a bandwidth waste for the forum's or poster's server...

[wkitty42]

geekay,

you can block on blank referrers...

# this ruleset is to stop blank user agents with blank referrers
# they are, however, allowed to access /robots.txt
RewriteCond %{HTTP_REFERER} ^-?$
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REQUEST_URI}!^/robots\.txt$[NC]
# let webmasterworld.com thru by name and/or ip
RewriteCond %{REMOTE_ADDR}!^westhost32\.westhost\.net$[NC]
RewriteCond %{REMOTE_ADDR}!^216\.71\.84\.181$
RewriteRule .* /cgi-bin/noagent.cmd [L,T=application/x-httpd-cgi]

as for your comment about wasting the poster's bandwidth, i guess you'd rather let them waste /your/ bandwidth and steal from you? consider if you are on a hosted setup and you have to pay x$ a month for the bandwidth your servers use ;)

[geekay]

While it is technically possible, blocking empty referrers requires careful consideration due to the drawbacks.

Today I see four forums with active threads hot-linking to my images, and those threads are unbelievably popular. To my regret I have therefore now had to disallow hot-linking by Google Images, because that was where the rogues found the illustrations for their posts.

For the benefit of "good" users I had exempted hot-linking done by Google Images. Now GI will still display the thumbnails, but the full-size images will no longer open. Next step will perhaps be to disallow the GI bot from my sites.

With an increasing number of users browsing with their referrer disabled this bandwidth issue can no longer be kept under control with simple means like mod_rewrite and serving a 1x1 pixel pic. I am also plagued with new 302 redirects (which at least MSN Search still cannot handle). My info sites seem to be useful.

[jdMorgan]

The code that wkitty42 posted blocks on blank or hyphen (spoofed blank) referrers only if the User-agent is also blank. Access is allowed to robots.txt on principle, and also to allow testing with WebmasterWorld tools.

I'd also suggest adding an exception for HEAD requests, since I've seen AOL's cache do this often since I posted the original version:

Another exception would be for favicon.ico requests, which MSIE6 seems to request without a referrer or UA.

Actually, I block blank referrers and spoofed blank referrers separately. I allow access if the referrer OR the User-agent is blank, but not both. And if either the referrer or the User-agent is spoofed blank (by using a hyphen as the string in an attempt to get past blank referrer blocks) then I'll block those requests. After all this typing, I suppose I should just post the code... ;)

# BLOCK *Faked* blank referer -OR- UA (client actually sends a hyphen, so as to appear to be blank in logs)
RewriteCond %{HTTP_REFERER}<->%{HTTP_USER_AGENT} ^-<->¦<->-$
RewriteRule .* - [F]
#
# BLOCK blank referer -AND- UA except for HEAD (and favicon for MSIE6 Win NT5.1 SV1)
RewriteCond $1 !^favicon\.ico$
RewriteCond %{REQUEST_METHOD} !^HEAD$
RewriteCond %{REMOTE_ADDR} !^216\.71\.84\.181$
RewriteCond %{HTTP_REFERER}<->%{HTTP_USER_AGENT} ^<->$
RewriteRule (.*) - [F]

This uses a neat trick originally posted by Andreas_Friedrich that concatenates two or more variables so that they can be tested by one RewriteCond. Note that the use of "<->" is arbitrary -- it has no special meaning to mod_rewrite or regular expressions. It is used to imply concatenation to a human reader, and also serves to define the boundaries between the variables, so that the regex parser can isolate the end of the first var and the beginning of the second. You could just as well use any rarely-occurring string to do this. You must be careful using this technique, since it can make the "anchoring" quite ambiguous, however, it works well for this application.

But the facts remain that you shouldn't block on blank referrer alone unless you can afford to lose a lot of users who are behind corporate or ISP caching proxies, and that serious protection against bandwidth leeching requires a script/cookies solution.

Replace the broken pipe "¦" characters with solid pipes before use.

Jim