Forum Moderators: phranque

Message Too Old, No Replies

Preventing Hotlinking

If referrer isn't this site, then no images for you....

         

g1smd

9:39 pm on Oct 16, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



OK. Some simple code for preventing hotlinking, but which line: domain1, domain2, or domain3, is the best code for this?

Each line tests for the domain at both www and non-www and on both .com and .co.uk, but which code is optimum, faster, preferred, less side effects, etc? Is it domain1, domain2, or domain3?

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER}
!^http://(www\.)?domain1\.(com¦co\.uk)/.*$ [NC]
RewriteCond %{HTTP_REFERER}
!^http://(www\.)?domain2\.com?(\.uk)?/.*$ [NC]
RewriteCond %{HTTP_REFERER}
!^http://(www\.)?domain3\.co(m¦\.uk)/.*$ [NC]
RewriteRule \.(gif¦jpg¦jpeg¦png)$ - [F,NC]

Fixed typo.

[edited by: g1smd at 10:02 pm (utc) on Oct. 16, 2006]

jdMorgan

9:53 pm on Oct 16, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I'll take door #2, with tweaks to each line:

RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} [i][/i]!^http://(www\.)?domain2\.com?(\.uk)? [NC]
RewriteRule \.(gif¦jpe?g¦png)$ - [NC,F]

"." is equivalent to NOT blank, since it will match any character if one ore more is present.

Escape the literal periods.

"jpg¦jpeg" replaced by "jpe?g"

Whenever you see "^.*" or ".*$", it's a good bet that you can simply delete all three characters, since an un-anchored pattern is equivalent. The only exception is where you wish to make use of the fact that ".*" is the 'greediest' pattern, and you are trying to keep the adjacent match as short as possible.

Change all broken pipe characters "¦" above to solid pipes before use; Posting on this forum modifies the pipe characters.

Jim

[edited by: jdMorgan at 9:53 pm (utc) on Oct. 16, 2006]

g1smd

10:01 pm on Oct 16, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hmmm. I just noticed that my domain2 code would allow someone to register domain.co in Cuba, and then hotlink to you and be served the images; so there is a loop-hole in that one.

If the domain test is unanchored, does that also mean that your code also allows domain2.co.jp and domain2.com.au and so on to match too?

.

Is it possible to over-ride the block to hotlinking for just ONE image on a site?

It is a logo that other sites sometimes display as a clickable link.

I assume put it in a separate folder and have a different ruleset for just that folder.

How do you over-ride a hotlink ban?

jdMorgan

10:22 pm on Oct 16, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I should also note that in a benchmark test [webmasterworld.com] (msg #441898) reported by andreasfriedrich several years ago, he found that using inline regex ORs (a¦b) inside RewriteConds was faster in .htaccess (which is interpreted at runtime), while using separate RewriteConds with [OR] flags was faster if the code was in a server config file such as httpd.conf (which is compiled at server startup).

Jim

jdMorgan

10:31 pm on Oct 16, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Well, to prevent the .co problem, you could revert to an explicit TLD pattern. Allowing one image to be hotlinked unconditionally just adds a RewriteCond:

RewriteCond %{REQUEST_URI} !^/images/logo\.gif$
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://(www\.)?domain2\.(com¦co\.uk) [NC]
RewriteRule \.(gif¦jpe?g¦png)$ - [NC,F]

or alternately:

RewriteCond $1 !^images/logo\.gif$
RewriteCond %{HTTP_REFERER} .
RewriteCond %{HTTP_REFERER} !^http://(www\.)?domain2\.(com¦co\.uk) [NC]
RewriteRule ^([^.]+)\.(gif¦jpe?g¦png)$ - [NC,F]

But don't ask me which is faster -- I don't know... :) I suspect the first version, but I'm not sure.

Change all broken pipe characters "¦" above to solid pipes before use; Posting on this forum modifies the pipe characters.

Jim

g1smd

11:23 pm on Oct 16, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The first example works fine.

I don't even know how to begin to decipher how the second example works.

I made a lot of progress today; and that finishes it off.

jdMorgan

1:52 am on Oct 17, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The only difference is that in the second example, the first RewriteCond back-references the image name captured in the RewriteRule as $1. I didn't quite code that correctly, though, because the pattern in the second rule above should be:

^([^.]+\.(gif¦jpe?g¦png))$

Note that the closing paren for the outer set of parentheses (back-referenced as $1) was misplaced.

That's part of the beauty of the way rules are processed: RewriteRules can back-reference RewriteCond variables, and RewriteConds can back-reference RewriteRule variables. Now if only mod_rewrite would add a "compare-two-variables" function, I'd be perfectly content. But alas, it's not directly supported in mod-rewrite, and can be done only on some servers whose regex libraries support 'atomic' back-references -- FreeBSD with POSIX 1003.2 support, for example, by using the fact that if A+A=A+B, then A+B.

Jim

g1smd

2:00 am on Oct 17, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Pfffft. I'll look at that again tomorrow.

At 3a.m. it isn't making much sense right now.

Thanks again.

g1smd

12:52 am on Oct 22, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I tried this, so that if the referer is one particular site, the image is replaced with some other image:

RewriteCond %{HTTP_REFERER} ^.*other\.site.*$ [NC]
RewriteRule \.(gif¦jpg¦jpeg)$ http://www\.this\.site\.com/replacement\.$1 [R]

I cannot get it to work. I have simplified it. I even made the test for just a particular full URL referer. I made it JPG only. Whatever I did, it never worked.

jdMorgan

1:01 am on Oct 22, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Ah, but what is to stop it from looping? :)

Add a RewriteCond to exclude /replacement images from being rewritten/redirected.

Jim

g1smd

4:24 pm on Oct 22, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I assumed it would finish after one go.

Will this do it?

RewriteCond %{REQUEST_URI} !^/replacement.*$

Is it {REQUEST_URI} that is invoked here, or something else?

jdMorgan

4:43 pm on Oct 22, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Asuming it's always /replacement.gif or replacement.jpg, etc., just use

RewriteCond %{REQUEST_URI} !^/replacement\.

Anchored ".*" patterns like "^.*" and ".*$" in the absense of any other greedy subpatterns are a total waste of time and space... Just omit them.

Jim

[edited by: jdMorgan at 4:58 pm (utc) on Oct. 22, 2006]

g1smd

5:02 pm on Oct 22, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




Finally, it works.

.

Now to see if it combines with the other rules that serve all other hotlinking sites a 404 when they ask for any image as a hotlink.

g1smd

6:10 pm on Oct 22, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Grrrr. It worked just the once, and then never again.

g1smd

11:49 pm on Oct 24, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Like I said, I haven't a clue what I am doing:

RewriteCond %{REQUEST_URI} !^/images/hotlink [NC]
RewriteCond %{HTTP_REFERER} ^http://.*that-?site [NC]
RewriteRule \.(gif¦jpe?g)$
http://www\.domain\.com/images/hotlink/replacement\.$1 [L]

Not working.

jdMorgan

12:04 am on Oct 25, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The substitution URL is not a regular-expressions pattern, it is a literal string. Therefore, you should not try to escape periods (or anything else) in the substitution URL.

Jim

g1smd

12:13 am on Oct 25, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I don't understand that at all. Are you saying to use this?

RewriteCond %{REQUEST_URI} !^/images/hotlink [NC]
RewriteCond %{HTTP_REFERER} ^http://.*that-?site [NC]
RewriteRule \.(gif¦jpe?g)$ http://www.domain.com/images/hotlink/replacement.$1 [NC,L]

If the URL has to be a literal string, does that mean I cannot use the $1 in it too?

jdMorgan

12:33 am on Oct 25, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



> I don't understand that at all. Are you saying to use this?

Yes, and substitution is a defined term in the Apache mod_rewrite docs.

> If the URL has to be a literal string, does that mean I cannot use the $1 in it too?

No, it doesn't, because $1 through $9, %1 through %9, and any valid %{SERVER_VARIABLE} or ${REWRITE_MAPS} will be substituted into that substitution string (whether that's what you intended or not) before it is used.

Note that the argument on the right side of a RewriteCond is (usually) a regex pattern, while the argument on the left side of a RewriteRule is always a regex pattern.

Jim

g1smd

12:40 am on Oct 25, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



So does that mean that the bolded \ in this folder rewrite have to be deleted too?

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /gadgets/widgets/(abc¦fgh¦rst¦xyz).*\ HTTP/
RewriteRule ^(([^/]*/)*)(abc¦fgh¦rst¦xyz)\.widgets(.*)$ http://www\.domain\.com/$1$3$4 [R=301,L]

Isn't that a literal string too? I'm confused, 'cus that one works.

jdMorgan

12:49 am on Oct 25, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, no slashes in the substitution.

[added]You might also want to specify R=301 or R=302 in your new rule that is causing you trouble. [/added]

Jim

[edited by: jdMorgan at 12:51 am (utc) on Oct. 25, 2006]

g1smd

8:05 pm on Oct 25, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hmm, I don't want to redirect, I want to show alternative content when one particular referrer is on any image requests. It's just an internal rewrite, no?

jdMorgan

4:04 am on Oct 26, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That's problematic:

For efficiency, you want your images marked as cacheable with long expiry times.

But if it is at all possible that a visitor might fetch your image from a hotlink, and then subsequently visit your site to see the real image, then he'd still have the cached alternative image in his browser cache. So, he'd see the alternative image on your site as well, because an internal rewrite won't change the URL, and therefore his browser will use its cached copy.

So this is a relatively rare case where a 302 is often the best response. Set the hotlink alternate image with a short expiry time and/or mark it uncacheable, while leaving your real image cacheable with a long expiry time. That way, if the vistor does come to your site, the alternate image will be replaced by the real image.

Frankly, I usually just serve a 403 response and don't otherwise bother...

If you do want an internal rewrite, then don't include http://www.example.com in the substitution URl, and don't use the [R] flags.

Jim

g1smd

6:47 pm on Oct 26, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



One of the uses of this "replacement" image is for blocking the hotlinking of people selling secondhand kit on ebay, who are linking to the main distributors images as hotlinks and draining vast amounts of bandwidth.

They don't mind if people host the images elsewhere, they aren't going to follow through on copyright violations as the images belong to the manufacturer, but the distributor is seeing 50% of their bandwidth being sucked by e-bay hotlinking.

g1smd

12:40 pm on Oct 27, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Having spent more than 10 hours on it, I still cannot get it working.
I just don't know enough, to be able to see what the problem is.

jdMorgan

1:28 pm on Oct 27, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Are you flushing your browser cache before each and every URL test? If not, then your browser will keep the last response it got --successful or unsuccessful-- until the expiry time given by your image's Expires and Cache-Control headers.

If that doesn't help, we'll need more details on precisely *how* it doesn't work:

  • How did you test?
  • What was the result?
  • How did that result differ from your expectations?

    Jim

  • g1smd

    3:10 pm on Oct 27, 2006 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



    I just get a 500 Error sent for each of the images, whenever a hotlinker requests them.

    I'm using Live HTTPheaders to see what is going on, and getting nowhere.

    g1smd

    4:06 pm on Oct 27, 2006 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



    OK. Now I can get it to 302 redirect to another URL, and then it continues redirecting to itself until the server crashes. Hmmm.

    jdMorgan

    4:18 pm on Oct 27, 2006 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member



    What code are you using now?

    We already discussed preventing a loop above -- Is the URL-path in your replacement image exclusion RewriteCond correct?

    Jim

    g1smd

    4:30 pm on Oct 27, 2006 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



    Aaaargh. It was a problem with my test server configuration.

    Moved the whole lot to another server, and it worked first time.

    g1smd

    4:36 pm on Oct 27, 2006 (gmt 0)

    WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



    Now to mail this code off to the site owner to install on the real site:

    RewriteCond %{REQUEST_URI} !^/images/hotlink [NC]
    RewriteCond %{HTTP_REFERER} ^http://(.*)e-?bay [NC]
    RewriteRule \.(gif¦jpe?g)$ /images/hotlink/replacement.$1 [L]