What causes the cs(Referer) to be blank?

Bookmarks, e-mail links, and privacy settings?

[nonprof webguy]

Hi folks,

I'm trying to explain to my colleagues why we only have referer information for about half of our "human" visitors and I realize that I don't actually know the answer!

I screen out user agents that are most likely to be spiders/robots, etc, but I know there are surely some UA fakers that sneak through. Nevertheless, I still have about 50% of unique visitors who leave no referer on their first hit on my site and just a "-" mark is recorded. Interestingly, that number has declined steadily from almost 80% 12 months ago.

I am guessing that some folks use browsers or browser add-ons that hide this info from me, which I don't mind. But what are some of the other reasons why UAs don't leave any referer? Bookmarks? Email links? Type-ins?

I'm also wondering if others among you have similar rates of no referers. Is 50% high? Low? Is it a sign I'm not screening robots well enough?

Thanks in advance to any insight into this phenomenon.

[aaronc]

Mozilla blocks referers as default.
Also, if someone types in the url directly, it doesn't show a referer.
I think clicks from email clients don't always show a referer either.

[aaronc]

Coming back to reread my post.
I meant to say Opera, not Mozilla.
Opera blocks referrer by default.
Some versions of Mozilla didn't pass referrers.

[Mikkel Svendsen]

Some companies filter out referrers and other "personal info" from the http request in the firewall.

Are you sure you have all agents filtered out? There are MANY off line browsers and e-mail harvesters that use a common browser agent names. They can be really difficult to filter out :)

[nonprof webguy]

Thanks for the tip about Opera. Right now, Opera is so little-used that I doubt it could be responsible for that many of my "-" referers.

But the firewall scrubbing makes a lot of sense. We have a large portion of our audience coming from government, and I suspect they are more likely to be behind firewalls that would leave these columns in my log blank. I'll have to do a cursory check on some of their IPs to see if there's a pattern.

I'm definitely not sure I'm screening out all robots, but I try to catch the most significant fakers by screening out any UAs that have a lot of hits in a short time, i.e. spider behavior. FYI, for anyone who's interested, my latest list of robot user agent strings (804 total) is in this text file [johnmccrory.com].

[gsx]

Norton Personal Firewall hides referrer. Lots of people use this.