Referers in Windows logs

[paul12345]

I have been looking at my site's logs. I am (shared) hosted on a Windows server. The last aspect of each individual posting in a Windows log is the "referer".

In a fair number instances there simply is no referer listed. The visitor in each of these instances seems to be doing the normal visitor things, but every time their visit access a new .htm or .gif no referer address is given. Why?

[DanA]

Sending a "referer" can be blocked by some firewalls, optionally blocked by the browser (Mozilla for example) or a plugin, not sent by some browsers... for anonymous browsing or any other reason,
while it may be useful for the webmaster or necessary for browsing...

[paul12345]

My concern with all of this stemmed from trying to identify hotlinking of my site's images by other sites. I was looking to see what the refering page was when my graphics were accessed.

Am I right in thinking that for those log entries where no referer is listed that this is an option that an individual site visitor has chosen (related to the way they have their browser options configured) and is not related to some method by which a site hotlinking my images has cloaked their activities?

[jdMorgan]

paul12345,

It's more likely a proxy between the user and your site. In cases where there is never a referer for any page, that may be a setting in the user's Internet Security software (or similar), but again the user may not be aware of it.

The site that hotlinks to you can't effect the referer sent by the user's browser - the user's browser request for the image is a new and a separate HTTP request -- a new "HTTP session" so to speak -- and not related to their access of the site that hotlinks to you. The only thing that gets passed from that request to the new request is the URL of your image.

In many cases where an intervening proxy is dropping the referer, you'll also see the user agent change, often to "Mozilla/3.01 (compatible;)". Also, you'll see AOL users (identified by their IP address) with no referer and no user-agent doing HEAD requests. This is the AOL cache checking to see if the document has been updated (AOL just has to be different, so they use HEAD instead of a GET with IF-MODIFIED-SINCE).

Best practices are usually to allow referrals from your own domain, blank referrers, and referrers from your site's IP address if you have a non-shared IP address. If you block blank referers, you may drown in customer-service calls and e-mails because your site will look broken to many users behind ISP and corporate proxies, or to those who use Internet Security software or have configured their firewalls for tight restrictions. It's unfortuante, but HTTP_REFERER is very unreliable. :(

Jim