Welcome to WebmasterWorld Guest from 107.23.176.162

Forum Moderators: phranque

Message Too Old, No Replies

Thwarting hotlinkers is great fun

My thanks to you... and another question

     
8:58 pm on Apr 3, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
posts:66
votes: 0


Many thanks to everyone here who has posted so much valuable info about using mod_rewrite to block hotlinkers. Initially I set my htaccess file in specific graphics folders to merely block them without returning a substitute image. However, I have a couple of graphics sites and apparently some of my images have become popular for use in forums - I found that many of the hotlinkers were not removing their links just because the images weren't showing up - so I was still losing bandwidth to these thieves every time the pages were viewed. I changed my htaccess file to substitute a nicely worded thief image and the result is that these folks have begun to remove the link coding rather than look at my image. :)

However, in one section I have certain images in a mysql database and I was surprised to discover thieves were linking to the php script that calls the images. I hadn't protected the directory holding the php script - so I used the following:

RewriteEngine on
RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://(www\.)?mydomain.com/.*$ [NC]
RewriteRule ^.*$ [mydomain...] [R,L]

I'm thrilled to say this seems to be working!
Having said all that I have another situation and I'm unsure of the proper way to deal with it. I'm about to open a photo gallery powered by php/mysql. All the pages/images have incredibly long dynamic urls. I don't want anyone to be able to link directly to the photos or any of the gallery categories and I want visitors to enter the photo gallery only from within my site.

I've been scouring the posts about both mod_rewrite and SetEnvIf Referer and can't seem to get my head around the right approach. In addition to preventing hotlinking of the images, I want to prevent visitors from arriving in the 'middle' of the gallery - I want them only to come from another page on my site. Hope that makes sense?

Hope some of you experts can give me some advice on the best method for handling this - there are oodles of posts about this and I've read most of them (I think)... maybe I've read so much that I've confused myself. For what I would like to accomplish is one method more effective than the other? Thank you!

8:22 pm on Apr 29, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 11, 2001
posts:44
votes: 0


I just got my first taste of bandwidth theives today. I check my logs monthly (using analog first), and last month I noticed a large number of referrers from a chinese site. I "knew" that for the past month there was a few chinese sites on "widgets" that linked to my site bringing in vistors, I even had a few emails from visitors from china so I thought nothing about it.

The problem is that that particular sites that i was getting a whole lot of hits was a forum, but i just assumed that they were forums discussing my site. Yeah on hindsight that was very dumb.

This month I suddenly realised that this didnt really make sense. The forum linking to mine were the second biggest refferer after google for 2 months in a row! Suddenly, something clicked.

I opened my logs, searched for the url, and then checked what they were accessing. sure enough it was a gif file I was using. Some idiot decided to link directly to my images for use as his avatar.

I feel so stupid...

It's strange how angry i felt even though I wasnt really hurt. I was thinking of doing some nasty tricks, but in the end I decided to just delete the file. The irony is, the pic wasnt meant to be there anyway, I just uploaded it for a friend to download, and forgot to delete it.

Does anyone know a way to prevent such tricks? My server is MS's IIS so I can't use the .htaccess method. I can't use PHP either.

PS Looking at weblogs sure makes this webmastering business exciting :)

8:56 pm on Apr 29, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Oct 22, 2002
posts:260
votes: 0


Besides blocking them, you could use that link to your advantage. Divert them to an image of your choosing.

In other words, find out what file they're linking to and make a minor change to the image to show your domain address. That way, if folks viewing the forum thread are interested, they've got a web address they can type in to get more information. A great and easy way to advertise your business without getting nasty with linkers who could bring potential customers to your website. :)

You can use coding something like what Busynut is showing above, but make sure you force gifs to gifs and jpgs to jpgs, or their could be some pretty weird results when the browsers misinterpret the image files.

I'm not sure of the specific coding that would work for this, as I'm only just starting to get into depth on such things, but I know it's possible and a search of WebmasterWorld should get you what you need - probably a couple of different variations on the theme, too.

Good luck!

12:22 am on Apr 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 1, 2002
posts:774
votes: 0


Busynut:

The solution to hot linking, you seem to have well in hand- that is great. Plus, it lets you "dip your toes" in the giant that is mod_rewrite!

I would consider using mod_rewrite to shorten the URL's to your images.... make them a bit more user friemdly. If you want to post sample URL's, and a rough idea of the actual path to your images, I am sure someone could take a crack at "fixing up" those long URL's...

>>> In addition to preventing hotlinking of the images, I want to prevent visitors from arriving in the 'middle' of the gallery - I want them only to come from another page on my site. Hope that makes sense?

This, again, could be fixed with mod_rewtite, based on referer. I am not sure if you want to, though. The idea would be to protect the interiour pages so if the referer was not your index page (or whatever), you mod_rewrite it TO the ibdex page. However, some browsers and some security programs (and proxies!) do not send referers, or accurate ones. You just might be cutting a lot of people off...

dave

10:09 am on Apr 30, 2003 (gmt 0)

New User

10+ Year Member

joined:Mar 26, 2003
posts:28
votes: 0


Carfac and Busynut:

I also block hotlinkers and send them a nice banner instaed of the desired image. Lot's of ads in lots of forums around the globe, I just love it! ;-))

To avoid blocking those visitors with browsers that don't deliver a referer I use the following line. As I said it's used in the htaccess part for the hotlinkers but my guess is, it'll work also if you redirect to the start page of your gallery or where ever.

RewriteCond %{HTTP_REFERER}!^$

This makes sure that blank referers are welcome too.

Anyhow, now I would like to ask something too: My problem is, that users come in through various misspellings like for example "http://news.mysite.com". There is no subdomain called "news" on my site, there's no subdomain at all. But people still get the html-files, just without the proper images but with my banner for each image instead. As I said, "wrong referers" are blocked.

Now I want to avoid these kind of misspellings and send them to only on spelling for my domain like "http:www.mysite.com".

My htaccess is quite big, lot's of rules and conditions, maybe too many for me to figure out this simple thing. Once I read a post where someone explained exactly that but I can't find it anymore. Could someone give me a little hint?

Thanks in advance and sorry for my bad english again and again ;-)

Konny

2:29 pm on Apr 30, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
posts:66
votes: 0


Hobbyist:
I'm fairly certain there is a method for accomplishing the same thing on an IIS server - hopefully someone with Windows experience can point you in the right direction. I would agree that Looking at weblogs sure makes this webmastering business exciting - but I also admit it is sometimes frustrating keeping up with it.

Syren_Song:

make sure you force gifs to gifs and jpgs to jpgs

I was concerned about doing this properly and in my case some images were being called from a script - so the url would look like this: "/folder/script.php?id=39"
I didn't know how else to do this so I just used the rewrite rule posted above - it appears to be working and is returning the substitute image (at least on the forums which I've been able to access). And bandwidth from these forums has decreased dramatically because they are now actually removing the link to the original image. What type of weird results would some browsers show?

Carfac and Konny:
I tried several different versions of a rewrite rule to prevent folks from entering in the middle of the gallery and none of them worked. So to be honest, I gave up. LOL. No matter how many posts and tutorials I read - my brain seems to want to block real comprehension of mod_rewrite, regex and htaccess. My photo gallery is actually just a personal project - if visitors to my site want to take a look they're welcome to but I have no desire for that section of my site (or the photos) to be listed in search engines. So for the moment I'm trying to rely on robots.txt and the robot trap script for a moderate level of protection. In any event, here's the last version I tried (which didn't work)

RewriteEngine On
Options +FollowSymlinks
RewriteCond %{HTTP_REFERER}!^$
RewriteCond %{HTTP_REFERER}!^http://(www\.)?mydomain.com/.*$ [NC]
RewriteRule ^/gallery/(.+) [mydomain.com...] [L]

As far as the long dynamic urls - I guess I've changed my attitude on that also. Since I'm not trying to make it easier for spiders or anyone other than actual visitors to find (or link to) the photo gallery I guess it makes sense (in a weird way) to let the urls remain the way they are. (crazy, eh?)

Konny:
I would be interested in learning more about misspellings and strange requests for pages that don't exist also. Very recently I had several requests like this (which resulted in 404 errors):
/page%20refused%20for%20storage%20%5bhttp:/www.mydomain.com/index.html

and this:
/folder/anotherfolder/000000

I have no idea what either of those is about - have never seen them before.

Thanks to everyone for sharing your knowledge!

2:41 pm on Apr 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 17, 2002
posts:2251
votes: 0


Hobbyist:
Google on HotLinkStop. It's an ISAPI filter for IIS and it works very well.
4:39 pm on Apr 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


KonnyQ,

Anyhow, now I would like to ask something too: My problem is, that users come in through various misspellings like for example "http://news.mysite.com". There is no subdomain called "news" on my site, there's no subdomain at all. But people still get the html-files, just without the proper images but with my banner for each image instead. As I said, "wrong referers" are blocked.

This should work:


RewriteCond %{HTTP_HOST} !^www\.yourdomain\.com
RewriteRule ^(.*)$ http://www.yourdomain.com/$1 [R=301,L]

This means "If the requested domain is not exactly www.yourdomain.com, redirect to www.yourdomain.com"

Having this redirect also simplifies many subsequent rewrites, since you will have one and only one valid referring domain to test for; www.yourdomain.com is the only possible valid referrer once a visitor has been redirected by the code above. So, for example, the second RewriteCond below could be simplified to:


RewriteCond %{HTTP_REFERER} !^http://www.mydomain.com

Since only that one referer would be possible if the visitor has already been redirected to that domain. This removes the requirement to repeatedly test for upper/lowercase and "www" and "www-less" variations.

Busynut,

Your problem can be fixed with mod_rewrite, but nowhere can I find a definition (URL structure) of "the middle of the gallery." Because mod_rewrite requires precisely defining regex patterns, no-one can write a rewriterule without a precise deifinition of what the required pattern match should be. You'll have to define the difference between a referer indicating "proper arrival" and one which indicates "improper arrival" in order to come up with a redirect that works.

Also, you can limit the load on your server by testing specifically for image-type files and using an internal redirect in the RewriteRule, and also get rid of inefficient regex redundancies like ".*$" by using the following:


RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?mydomain.com [NC]
RewriteRule \.(gif¦jpe?g)$ /thief.gif [L]

One other note: As I've said before, it is a very good idea to not redirect a browser from one file type to another. I imagine that most of the time it will work, but maybe not. The "cleanest" approach is to make a version of your hot-linking image in each format that you use on your site -- for the example here, one in gif format, one in jpeg format, and one in jpg format. Then redirect each hot-linked image to the correct matching filetype:


RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www\.)?mydomain.com [NC]
RewriteRule \.(gif¦jpe?g)$ /thief.$1 [L]

Much of the above is "details," but I'd like to leave clean code and examples for those who visit this thread later.

(As always, hand-edit the "¦" characters above to use the solid vertical pipe on your keyboard)

HTMS (hope this makes sense),
Jim

6:24 pm on Apr 30, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
posts:66
votes: 0


Thank you, as always, Jim, for the additional clarification. I know you end up repeating yourself hundreds of times attempting to help people like me whose brains are on permanent holiday.
it is a very good idea to not redirect a browser from one file type to another.

Creating substitute images for both gif and jpg files isn't a problem and what you and others have suggested definitely makes sense. I just didn't know how to create a substitute image for a url which is actually a php script(script.php?id=39). This is what several forum hotlinkers were using. This similar topic was discussed not that long ago concerning other file types (.wav, .mpg. etc.) - and I don't believe there was an acutal solution for preventing hotlinking to a variety of file types other than images [making note to do a proper search on this topic once again].

As far as preventing folks (or spiders) from arriving in the middle of my photo gallery, I've kind of given up on that idea anyway. I said earlier that the test rewrite rules I tried didn't work - but that wasn't entirely accurate. I caused an infinite loop for myself in one attempt - because I accessed the page without a referer - deleted that particular htaccess file pronto! I had hoped to allow anyone visiting a specific gallery page who had initially come through any page on my site (there is only one page on my site having a link to the gallery anyway); and anyone else arriving from any other site would be redirected to the front page of the gallery (mydomain.com/gallery/index.php).

Finally, I said a bit earlier that I wasn't troubled by the extremely long urls generated by the gallery script. Upon closer inspection of my log files I'm not sure that's the right attitude to have. These long urls are making my log files a lot bigger. Would using mod_rewrite to shorten the urls make any difference in the log files? ...meaning, if the long url was accessed and then redirected wouldn't the long url still be showing up in the log file anyway? An example of a url generated by the gallery script is as follows (breaking it into several lines to prevent a wide screen):

[mydomain.com...]
a%3A12%3A%7Bi%3A0%3Bs%3A1%3A%220%22%3Bi%3A1%3Bs%3A20%3A%
22ee_photo.ee_photo_id%22%3Bi%3A2%3Bs%3A3%3A%22ASC%22%3Bi%
3A3%3Bs%3A20%3A%22ee_photo.ee_photo_id%22%3Bi%3A4%3Bs%3A0%
3A%22%22%3Bi%3A5%3Bs%3A1%3A%221%22%3Bi%3A6%3Bs%3A6%3A%22thumbs%22%3Bi%3A7%3Bs%3A1%3A%220%22%3Bi%3A8%3Bs%3A6%3A%
22public%22%3Bi%3A9%3Bs%3A7%3A%22default%22%3Bi%3A10%3Bs%
3A1%3A%227%22%3Bi%3A11%3Bs%3A2%3A%2210%22%3B%7D

Sorry to be so dense. I'm not at all sure I make any sense anymore :)

6:47 pm on Apr 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Busynut,

On the script: If you modify the script to do an external redirect (a 301) to the image, you can still filter by filetype. Better yet, modify the script itself to check the referrer, and allow only on-site (or blank) referrers to access the real images.

Your long URLs will still show in the logs if that is what users' browsers request to get to the resource. Again, the trick is to modify the script to output short spider- and user-friendly URLs, and then rewrite those to the longer version internally.

The infinite loop problem can be fixed by adding a RewriteCond that prevents the redirect if you have already done it and are now coming from an "allowed" referrer. Again, the devil's in the details...

HTH,
Jim

2:42 am on May 1, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
posts:66
votes: 0


Hey there busyJim,

Better yet, modify the script itself to check the referrer

That's probably the best suggestion yet in all honesty. Get it at the source. (scheduling time to study php scripting) I don't know a dern thing about programming but how hard can it be to find out how to add coding for this? Thanks for the idea, Jim.

About the long urls - I kinda knew that would be the answer. They'll still be in the logs using up space so there's no point in trying to rewrite them.

again, the devil's in the details...

Truer words have never been spoken. I hate details! LOL

Thanks again!

11:31 am on May 1, 2003 (gmt 0)

New User

10+ Year Member

joined:Mar 26, 2003
posts:28
votes: 0


@Jim

That's exactly what I needed, thanks again for your help, you really made my may. ;-)

@Busynut

"people like me whose brains are on permanent holiday."

Guess I know what you're talking about .. ;-))

KonnyQ

5:31 pm on May 8, 2003 (gmt 0)

New User

10+ Year Member

joined:Mar 4, 2003
posts:18
votes: 0


Busynut,

To stop people from entering in the middle of your gallery you just need to set a cookie. All your pages will check for that cookie and if they don't have the cookie then they will be redirected to your main page which will set the cookie.
9:37 pm on May 14, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 17, 2003
posts:1947
votes: 0


I have some funny videos on my site that people link to directly all the time (especially on message boards). At first I thought this was altogether bad but then I thought, "I'll just change the name of the folder occassionally and send 404's to the videos page that has the videos". I THINK this results in a lot more actual page hits than preventing hotlinks altogether (i.e., a redirect to the videos page is better than a "access denied" message).

Fun!

12:07 am on May 13, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 23, 2002
posts:66
votes: 0


Welcome to Webmasterworld, Pigsfeet, and many thanks for your suggestion - will be testing that out soon. And apologies for taking so long to acknowledge your post - haven't had much time to get in here lately!

HughMungus,
I think you and I are doing the same thing with these forum hotlinkers, but I'm doing it with my htaccess file (as posted) and I don't have to change the name of the files or folders. The substitute image the hotlinkers are getting says: "the image that was supposed to be here has been hotlinked by a bandwidth thief... have a really nice day!... mydomain.com" I get a kick out of seeing it (if only briefly!) before the hotlinkers remove their link.

BTW... both your names make me smile ... hehe... would love to know the stories behind some of the user names in here! :)

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members