Is it safe to block hotlinking when there is no referrer?

[Sgt_Kickaxe]

Google's new image search layout no longer sends a referrer when hotlinking your image, it used to. The result is that Google now shows your image hotlinked on their site instead of loading a cached copy on their own server.

This isn't ideal since scrapers generally grab the url of the image they are stealing and I'd much prefer they grab Google's cached url instead of my site's url.

Is it safe to go ahead and block images from displaying if there is no referrer? A blank referrer can happen in some situations such as when behind a company firewall.

I'm asking this from an SEO standpoint, not about code, but here is the htaccess that will be left if the blank referrer check is removed.

rewritecond %{HTTP_REFERER} !^http://(www\.)?example\.com [NC]
rewriterule \.(gif|jpe?g|png)$ - [NC,F]

[lucy24]

"No referer" includes:

99% of all search engines, good or bad
and
some {browsers | hosts | proxies} with or without the human user's knowledge or consent
and
anyone, anywhere, pasting the image URL directly into their address bar, or using a bookmark

So it depends what you mean by "safe". You can choose to make exceptions for known quantities, such as your favorite search engine. But it is well-nigh impossible to block all referer-less requests without preventing some real humans from seeing your pictures.

Rather than [F] I'd rewrite to something like a single-pixel gif. (A "No Hotlinks" or "Stop Thief!" image would be unfair, because some of your affected humans don't know they're not sending a referer.)

Either way, make sure your page code includes explicit height and width declarations for all foreground images, so overall page layout isn't affected.

[Sgt_Kickaxe]

How would you handle Google images then Lucy, assuming you wanted/needed to get them to stop hotlinking images?

They no longer pass a referrer from their search results pages in image search but their image-search bot still provides one.

[keyplyr]

I have always allowed a blank referrer in my anti-hotlinkg code. Employees surfing the web from behind their company firewall quite often send a blank referrer, as do many schools, libraries, internet cafes, military and several mobile networks. Don't want to weird-out all those visitors & potential customers.

[Sgt_Kickaxe]

So is it impossible to stop Google from hotlinking your images without affecting your visitors?

Can you safely block just Google image serps without stopping them from ranking images?

I know that sounds counter-intuitive but my niche leaves my images especially prone to scrapers who find them in search and they link them from spammy sites. I'd really like Google to use their own cache copy instead.

[LunaC]

Would this be cookie territory?

eg.
Stage 1:
Use regular hotlinking code in .htaccess but allowing blank referrers

Stage 2:
Once they land on the page, test if the real page is loaded on your site by a real visitor using a cookie. If the cookie is there, show images. If no referrer and no cookie, block / redirect?

I know nothing about cookies or if this is even possible, just thinking off the top of my head for possible solutions.

[incrediBILL]

How would you handle Google images then Lucy, assuming you wanted/needed to get them to stop hotlinking images?

Block them in robots.txt from indexing your images in the first place.

If you don't want hotlinking from a search engine, why let them index the images?

[lucy24]

Block them in robots.txt from indexing your images in the first place.

You mean "block them from seeing your images", right?

Interesting hypothetical question there. If they know the image exists, and they can read its alt and the nearby text, will it show up in image searches?

Cursory investigation --looking for images from my ebooks directory, which has been off limits to the imagebot for ages-- suggests that a robot block alone may really work. At least sometimes.

If you don't want hotlinking from a search engine, why let them index the images?

Uhmm... Because you want the search engine to show a cached thumbnail, which in turn will lead people to your own site? Seriously now, isn't that like saying "If you don't want the search engine to show a preview, why let your page be indexed"?

[keyplyr]

Uhmm... Because you want the search engine to show a cached thumbnail, which in turn will lead people to your own site?

What top level SE bots still does that? Hasn't that been entirely replaced by snapshots?

[incrediBILL]

@lucy24 I think you misunderstood my comment. I was basically saying if you allow them to index then whatever else they do with the image, including hotlinking, is kind of fair game as long as it's used to promote your image. In for a dime, in for a dollar.

FWIW, I don't allow any images to be displayed unless the referrer is my own domain and haven't for years. Blank referrers are always banned and will always be banned. Period.

I have about 40K images on the server, screen shots to be exact, and needless to say there were more than a few sites attempting to heavily leech those images.

[matrix_jan]

I just blocked all direct (no referrer) requests besides some known bots.

[lucy24]

Hasn't that been entirely replaced by snapshots?

Snapshot, thumbnail, whatever you call it so long as it's cached.

if you allow them to index then whatever else they do with the image, including hotlinking, is kind of fair game as long as it's used to promote your image.

Now, wait. What does "promote your image" mean? From the webmaster's POV, image search means: Find sites that include pictures of X. And then the searcher goes to your site.

Only in the rarest case would you want it to mean simply: Find pictures of X in isolation. ("I found a photo of {famous person} doing {silly or illegal act} and I want the entire world to see it by any means possible.")

The still-unanswered question is: What proportion of image searchers fall into this second group? They don't intend to go to any site, they just want to look at pictures. Those are the ones you can safely block-- but only if you can do it without also blocking the first group.

[keyplyr]

Snapshot, thumbnail, whatever you call it so long as it's cached.

The point is IMO, you don't need to allow them indexing privileges to your image files for them to get a snapshot/preview of your site any longer... at least not from the tests I did last year and I see no problems to date.

I just blocked all direct (no referrer) requests besides some known bots.

Then you are blocking a huge amount of valid users. Almost all browsers can turn off referrers nowadays, plus the list I gave in one of my posts above.

[lucy24]

Google Preview is an entirely different issue.* What if your site depends on visitors who need sites-with-pictures-of-X? Ordinary humans don't use Preview for that; they use image search.


* Raising questions like "Why, exactly, are they loading .midi files when Preview does not have a 'play sounds' option?" to say nothing of the whole executing-javascript issue. Different thread.

[incrediBILL]

Then you are blocking a huge amount of valid users. Almost all browsers can turn off referrers nowadays, plus the list I gave in one of my posts above.

Except most users don't know what a referrer is and barely understand a COOKIE. The mass populations often doesn't even know what an Internet Explorer or a Firefox is and thinks their Google start page is the internet.

I've been blocking images for blank referrers for many years without any complaints.

[keyplyr]

Google Preview is an entirely different issue.* What if your site depends on visitors who need sites-with-pictures-of-X? Ordinary humans don't use Preview for that; they use image search.

I stop Google from getting my images. Google Preview shows a snapshot of my pages (including the images that are on that page) to the right of my listing in the SERP, so obviously it doesn't need to index my image files in order to produce the snapshot.

Except most users don't know what a referrer is and barely understand a COOKIE. The mass populations often doesn't even know what an Internet Explorer or a Firefox is and thinks their Google start page is the internet.

That scenario is changing. All browsers now have a private browsing setting (no referrer) or a stealth mode (same thing) in their tools, besides the employees behind firewalls, military, libraries, univ, internet cafes, several mobile networks, etc that don't send referrers.

Maybe your site looks good without images displaying for the visitor. I've seen my site without them, it looked awful.

I've been blocking images for blank referrers for many years without any complaints.

They won't complain, they just won't come back to a site that looks like that. I've come across plenty of broken sites. Unless I know the owner personally, I've never complained or reported it. Who knows, maybe they want it to look like that - LOL

Got an early class in the morning :)

[lucy24]

obviously it doesn't need to index my image files in order to produce the snapshot

Apples and oranges. The original post was a question about image search. I'm not sure how Preview even entered the discussion, since it has absolutely nothing at all whatsoever in any way-shape-or-form no way nohow to do with image indexing.

I think we are all agreed that Preview can only be kept out of files by brute force. The kind that has an [F] at the end.

[keyplyr]

The original post was a question about image search.

In a general way...

but the question was:

Is it safe to go ahead and block images from displaying if there is no referrer?

And IMO no, it's not safe (see examples above.) Then I went on to show that allowing images to be indexed is not necessary anyway so why let them be indexed in the first place? Disallow in robots.tx then use the Remove/Block URL tools in GWT and BWT. And I also appended an X-Robots-Tag: noindex to the header of all my images. Problem solved, no hot-linked images.

apples < > apples

[matrix_jan]

Then you are blocking a huge amount of valid users.

What percentage of users use browsers that don't send referrer data at all? 0.1%? 0.5%? And I'm talking about those cases when you visit a webpage and all the images, css, javascript files are being called as direct access... With my move I get 20% more traffic, so I don't care about that small percentage.

[tedster]

This referer issue has been around ever since browsers were created - and the specifics have changed but continue to show a of of weirdness. We had a discussion of this back in 2002, when Brett_Tabke summarized some of his research:

There are so many variations on referrer behavior from browsers, that if you are within 20-30% of reality you are doing good.

If your log file will allow you to do it, throw out everything but the first visit for any user. Only use those referrers. That will give you the most accurate account.

Other things that will throw off referrers:

- some browsers will only send the root domain for any site.
- some browsers and proxy servers will repeatidly send an external referrer for EVERY page it visits. If it comes in from Google, and they visit 20 pages, all 20 pages could see that same google referral string sent.
- Most clued in Opera users turn off referrals as a security precaution. Mozilla may have an option to do the same soon. They are arguing about it now.
- I have heard that there is a version of msn IE that will not report an external referral under some security settings (not sure, but the pattern fits).
- Revisits. If a page is reloaded, some browsers will sent that page itself as the referral. hence, the high proportion of www.mysite.com in your logs.
- no cache mania. Most of the dsl, cable, and other high speed modem manufactuers are telling people to turn off caching in their browser. They all have explicit details on their site as one of the setup steps to take. That in turn is skewing referral numbers as even a simple back button can cause a page reload. That referrer will often be the previous page.

It's been my experience that 50 to 75% of insite referrals are not correct. Bookmarks, typed-it-ins, drop down history from address bar, caching, no caching, and reloads have turned insite referral numbers to junk. There are no major log file analyzers that have this fact figured out.

[webmasterworld.com...]

[keyplyr]

What percentage of users use browsers that don't send referrer data at all?

All the current up-to-date major browsers can be set not to send a referrer, even my mobile web browser. Lots of misinformed people feel this is proactive security.

Don't know the exact percentage, and surely it's different for each site, but I see several dozen, maybe a couple hundred empty referrers from unique valid human visitors each day that I look for things like that. But I have no reason to keep track since I allow blank/empty referrers in my anti-hotlinking code.

[lucy24]

All the current up-to-date major browsers can be set ...

What a browser can do and what the human user does do are entirely different things. Especially if you think you're doing one thing and you're really doing another.

What happens when you select a menu option called Private Browsing? You may think it means that certain information isn't sent to the sites you visit, so megapage dot com doesn't learn any more about you than it has to. But in fact it means that certain information isn't recorded on your personal computer, so your spouse/ parent/ boss/ roommate can't come around later and snoop into your browsing history.

Or vice versa.

Meanwhile, for the users who happen not to have those "current up-to-date major browsers", or can't change the settings in the browsers they do have ...


* I have one friend who is reluctant to download Chrome-- not because it's a product of the Evil Empire but because he's afraid it will take too much of the computer's "memory", meaning disk space. (What he's saving it for is not perfectly clear to me.) I'm fairly certain he doesn't understand it's a browser, but thinks it's some kind of supercharged search engine. And I'm very certain this idea is not unique, even among people who actually use chrome. ("Oh, that's interesting, the Internet looks different now.")

I would not do well as a Technical Support person.

[keyplyr]

I don't see the point in arguing semantics.

Back to the OT.

A visitor uses their browser's bookmark/favorite to access your site, thus sending no referrer. You block empty referrers in you anti-hotlinking code (in some futile effort to stop Google from using your images) and the result is this visitor sees you site sans images, or some swapped image you've chosen. Who's the real looser here?

"cutting off your nose to spite your face" comes to mind.

[matrix_jan]

A visitor uses their browser's bookmark/favorite to access your site, thus sending no referrer.

Again, the page has no referrer but images do have and it's their parent page.

Well, the thing is that those who see my website with no images most likely might have not even visited the page if I had allowed hotlinking, because I get a lot of traffic when I block blank referrers(except from some bots). And from that point on it's their problem...

My guess is that 99% of internet users don't know how (and why) to block referral data. And the rest, 1%, are not "valuable" visitors because they're good in tech and rarely click on ads.

[keyplyr]

Did a count from yesterdays logs. Of the approx 12k in page loads, there were 576k requests. 3k requests had no referrer. Of those, over 2k were blocked by various IP, UA and Header filtering, leaving 1k assumed legit human overall requests. If I blocked referrers for images, that's a hell of a lot of users who would likely never come back and buy anything.

[edited by: keyplyr at 2:02 am (utc) on Feb 7, 2013]

[lucy24]

Pages and images are different things.

[keyplyr]

Why yes.... yes they are. We're not discussing blocking pages.

If you read the next sentence you'll see the overall request approx total, including image requests and other files attached to those web pages. Since I'm not the one blocking image requests, I have no interest in separating image file requests. I just know it would be bad for a lot of users.

[indyank]

Is it safe to block hotlinking when there is no referrer?

In a way he implied he didn't want any traffic (direct) for images from any of those search engines. He seems to be wanting traffic to his pages from all sources, but he wants the images to be available only to those visitors who come for his pages.

If my above understanding is correct, it would make sense for him to consider separating image file requests from web page requests.

[indyank]

@OP, have you tried blocking Google image bot via robots.txt? Doesn't it obey your robots.txt?

[matrix_jan]

@indyank

Personally I want all possible bots to index my images. I just don't want my images to get hotlinked.