apache logs and the "referrer" field

what causes unknown referrers?

[dataKris]

Hey Guys,

Again, this may not be a new topic to you trackers! but, please do help me out here.

I am sitting with apache logs trying to figure out where the 'unknown' referrers are. What causes apache to not log the referrer? Any heads up would be gratefully appreciated. Thanks.

[jatar_k]

a program could be blocking the referrer
could be a type in
bookmark maybe

I always hope for 80% accuracy and then I am happy.

[jdMorgan]

...user disable referrer option in Netscape/Mozilla/Firefox browser configuration (using about:config), clients like Windows Media Player don't ever send one, etc...

Design your access control to expect blank referrers and handle them gracefully.

Jim

[dataKris]

jatar_k,

Thanks for the quick response. I realized those could be the factors.. and have been happy with such rates..

But, the URLs that I am trying to track include unique params and are specifically designed for PPC.. ruling out type ins.. and leaving out only bookmarks and referral blockers..

the numbers of 'unknowns' I see are unbelievable to be hypothesized as bookmarks or blocked referrals..

I am just wondering if apache leaves out the referrals for any other reasons.. I don't know. Are any of you guys facing similar situations?

[jatar_k]

my tracking is done with php

I grab the referer and I parse the landing url as well. I append string like

?source=Gkeyword1

for all ppc campaigns. I grab that off of all my pages when they are hit. Between 5 and 7% of my referred traffic gets through with no referer but these keywords are still tacked on there so that increases my numbers as well.

[webdiversity]

dataKris,

If you've been doing PPC for a while then expect that many people will bookmark your tracking URL's thus creating the effect of being paid for clicks, when reality is you will only pay for one.

I'd suggest doing a lookup on the IP ranges to try to identify, if you have a lot of the same IP and you are doing PPC it pays to speak to the supplier (overture/google) about the specific publisher. We tend to be wary of these sorts of things because someone is trying to hide something, but that's not a blanket policy, many of the sources are good.

[dataKris]

jatar,

Actually, I do my tracking with PHP too.. the same way you described.

We started our PPC campaign about 8 weeks ago and I had the tracking set up and going.. but then, the number of 'unknowns' is almost 50%! - making it so hard for us to make any decisions.

I boiled down to four situations that might be causing this:
1. Bookmarks - I'll get to this.
2. Blocked referrals
3. Type In
4. My tracking script gets interrupted before it completes its logging - page is set up to allow that to happen.

To rule 4 out, I looked at the mother of all logs - the apache log - which seems to correspond to the log created with my script.. ruling out 4..

W.r.t bookmarks, the campaign has been ON only for a few weeks.. and I see 'unknowns' from some of these unique URLS.. which are impossible to be typed in and can only be referred by the SE's even to be bookmarked.. there are no links to the PPC pages from anywhere else.. the only options left are blocked referrals..

I see that maybe happening from both jdmorgan's post and urs.. but, such a high percentage? is that likely? Thanks guys.

[jatar_k]

its always likely

I had my script running for about 8 months with 70% effectiveness, split our secure and public servers and it dropped to 50%. I couldnt figure it out. I made a few changes, didn't help, a few more didn't help. I finally made a few more tweaks and it cruised up to 80 - 85% again.

Sometimes you just need to play with it. I also use specific landing pages for certain campaigns, that way all i have to do is grep the logs for occurences of that page.

We also filter our logs so that they are much easier to read.

a look at our filtering and some other PHP tracking tricks
Getting PHP session variables into your Apache logs [webmasterworld.com]