A new way of tracking blank referrals

A way of turning the blank referrals into useful information that can help.

[Ben_Graham]

Everyone who analyzes web logs will periodically look at their blank referrals and curse. While going through my bi-monthly shaking of my fist at the world, I came up with a realization that this information is actually very helpful.

In the past, I've always thought of blank referrals as meaning one of a number of things:

1. Spiders / bots
   
2. Users with referrals disabled
   
3. Users who access the site from a non-browser link (Outlook, e.g.)
   
4. Users who type their request into the address bar directly or use a bookmark
   

The problem was, you could never understand what the different percentages were. Were most of the users just ones who downloaded privacy software that was blocking everything? Or were they users who were using offline mail readers?

I use ClickTracks for my web analysis, because of its reasonable cost and good user interface. I then export the results to Excel for more interesting analysis. I currently define traffic as PPC if it comes in with a tag (source=) that I apply to all my advertising. The remainder is organic by default.

I analyzed a recent chunk of 100,000 visits, and found a very interesting fact:

• 14% of PPC traffic has blank referrals
  
• 38% of Organic traffic has blank referrals
  

What does this mean? Well, I think it's safe to assume that the users are going to be reasonably the same between two such large groups. If so, then the 14% would be the portion of users with referrals disabled. If that's the case, then the 24% additional blank referrals in the organics must be the users that are coming directly or through an email link.

If this is true (feel free to disagree) then the percentage of your traffic that's coming directly to you through favorites, email links, etc. is:

%Organics X (%Organicblank - %PPCblank)

Tracking this percentage over time should give insight into the real stickiness of your site.

Does anyone disagree? Does anyone have a higher percentage of blank referrals from their PPC traffic than their organic? That would certainly debunk my analysis.

[frugalsnow]

I just did a quick check of 100k click lines with status 200 or 304.

I got about 60% blank referers total.
And 19% of PPC clicks had blank referers.

I don't know if I agree with your analysis, yet.

[Ben_Graham]

frugalsnow, what are you using to calculate blank referrals? I've heard people claim up to 50% blank, but 60 sounds high.

[frugalsnow]

I use perl on the apache log files, and look for a referer with "-". This is after I've filtered out hits that aren't user clicks - i.e. .gifs, .icos, etc.

I just double checked - and yes, I have a log full of 100k clicks, and 60,000 have "-" for the referrer.

The reason that seems high may be due to the nature of the site - there are functions like knowledge base and support where referer may not get recorded.

[figment88]

I would think you would have to look for blank referals for pages not just any old objects. Most robots and automated tools don't request external css or javascript, many don't request images.

[Ben_Graham]

frugalsnow, are you excluding spiders from your counts?

[frugalsnow]

No I didn't exclude spiders... I'll see if I can do that later on.