homepage Welcome to WebmasterWorld Guest from 54.163.72.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Empty Referrer on 50% of logs
How many of yours are empty? Any solutions?
kapow




msg:904896
 1:19 pm on Aug 9, 2004 (gmt 0)

We're developing our own tracking system to gather return-on-investment intelligence. The system is going well, however, I am amazed at the quantity of visitors where the referrer string is empty ie no trackable referrer.

On average (over 3 popular websites) I find about 50% of logs have an empty referrer.

How many of yours are empty? Any solutions?

Note: we don't do any affiliate stuff its just natural and sponsored search engine visitors we're tracking.

 

Jon_King




msg:904897
 1:43 pm on Aug 9, 2004 (gmt 0)

I have the same problem. I've sent logs to Webtrends and our hosting service to no avail. Hopefully someone here has an answer.

webdiversity




msg:904898
 5:54 pm on Aug 9, 2004 (gmt 0)

This has always been an issue for us, albeit we operate in the PPC arena so the figures are probably more like 10% with no referrer.

I've always maintained that if the source of traffic is hidden then what lies beneath it, but as some of it converts to sales I can't be a total sceptic and assume it is all bad.

I think the PPC providers owe it to advertisers to ensure tracking abck to source, and encourage them to stop paying affiliates that deliver traffic with no source.

dataKris




msg:904899
 7:04 pm on Aug 9, 2004 (gmt 0)

My favorite topic:

Here are some previous posts that have discussed the same issue. Will post more later today:

[webmasterworld.com...]
[webmasterworld.com...]

dataK

ogletree




msg:904900
 7:07 pm on Aug 9, 2004 (gmt 0)

Many things can be happening. Some corp firewalls strip referrer logs. Norton strips it as well. Also I have found that most of my hits like that are from spiders. If you get a good spider trap and get a good .htaccess referrer ban list you can reduce that a lot.

kapow




msg:904901
 7:42 pm on Aug 9, 2004 (gmt 0)

most of my hits like that are from spiders

I'm not talking about just traffic, I'm talking about conversion tracking ie those people who bought something sometime (minutes or weeks) after visiting the site. We capture the referrer on first visit and then make checks when they buy (no need to go into all the details).

The problem is the initial referrer logs are empty for a lot purchasers! ie no referring website. As far as I can tell that means they didn't come from anywhere e.g. they typed the domain into their browser (which is possible as the sites are popular).

Or am I missing something?

roitracker




msg:904902
 10:40 pm on Aug 9, 2004 (gmt 0)

50% does seem pretty high, so there *could* be a problem with your cookie - then again, if you have lots of repeat visitors, 50% may be right.

For blank referrer info, read message 38 of this thread:
[webmasterworld.com...]

dataKris




msg:904903
 4:00 am on Aug 10, 2004 (gmt 0)

roi,
a link to a nice thread.. the post seems like my life story.

My biggest issue is with the amount of sales I have with no external referral. About 20% of the sales seem to always start at the addToCart page on my site. We kept theorizing it maybe due to bookmarks until we started doing PPC. Some of the links are so unique and so fresh, I can manually go through my logs and pick the IPs that visited those pages. There is no way those could have been bookmarked.

After I read through some postings, I realized what maybe happening. My product pages were getting cached and when the buyers attempted to add to cart or checkout, they were hitting my servers.. Further, a very random check of the whois info of the sale IPs show a major presence of AOL powered IPs.

I can't think of a single thing to do when my pages get cached.. I am almost sure that if I solve this, I would boost my accuracy by atleast 20%.

Any heads up would be appreciated.

digitalv




msg:904904
 4:15 am on Aug 10, 2004 (gmt 0)

the referrer string is empty ie no trackable referrer.

Many people either block the referring document through software (like Norton Internet Security, McAfee, a couple of Firefox plugins, etc.) and many others are surfing through anonymous proxies that do this automatically. Basing your tracking on the referring document is unreliable. I suggest you use a Tracking URL for your advertising sources. Rather than have your advertisements link to www.example.com have them link to www.example.com/?track=source

Then all you have to do is look in your logs for how many hits there were to /?track=whatever and you'll know how many clicks were delivered - you can ignore the referrer completely.

I can't think of a single thing to do when my pages get cached.. I am almost sure that if I solve this, I would boost my accuracy by atleast 20%

Subscribe to a hosted service (iwebtrack, webtrends live, etc.) and put the Javascript code in your pages. Even if the page is cached, the user will still be tracked because the Javascript will register a page view on the stat provider's servers every time. I use both the raw log files and a hosted service and between the two I get a much better idea of how many visitors there are and where they are coming from. But definitely use a tracking URL instead of the referrer when it comes to determining where your visitors came from.

dataKris




msg:904905
 5:10 am on Aug 10, 2004 (gmt 0)

digitalv:

I do appreciate your post. But, here are the issues.

I have my own javascript based logs and also look at the raw logs once in a well. Both show me the same situation - the log captures the IP only starting from the https pages or the dynamic addToCart page. Also, how could I explain the major presence of AOL IPs - a coincidence?

Thanks anyway.

cgrantski




msg:904906
 12:57 pm on Aug 10, 2004 (gmt 0)

Are we talking about the IP field or the referrer field?

kapow




msg:904907
 1:20 pm on Aug 10, 2004 (gmt 0)

*could* be a problem with your cookie

I don't think so, one of the sites has a shopping cart that depends on cookies ie you can't buy something unless a cookie is set. I am talking about ROI tracking ie tracking those that bought something, not just incomming traffic. When they arrive the system collects the referrer field, key words etc and stores it; when they buy the system adds that info to the data about purchasers. The problem is that up to 50% of purchasers have an originally empty referrer field. I go look at the raw logs (well my Programmer does) and see a lot of empty referrer fields.

use a Tracking URL for your advertising sources.

Yep, doing that. Our tracking system picks them up for Adwords and Overture, but also monitors non sponsored purchasers.

Are we talking about the IP field or the referrer field?

The referrer field.

kapow




msg:904908
 1:36 pm on Aug 10, 2004 (gmt 0)

Finally got round to reading message 38 of this thread: [webmasterworld.com...]

That is interesting! Some notes on it:

Re. 'Some browsers won't send a referral string'
I know there are lots of different browsers but like it or not I find 85%+ of visitors use IE.

Re. '...proxy server or other filtering agent...'
Surely thats a small minority. I'm talking about ordinary sites selling ordinary stuff to ordinary folk.

digitalv




msg:904909
 2:30 pm on Aug 10, 2004 (gmt 0)

I find 85%+ of visitors use IE.

Referrers are blocked in Internet Explorer more than any other browser - your own logs should prove that. As I mentioned before, programs like Norton Internet Security, some versions of McAfee, and lots of privacy/anti-spyware/adware software does this automatically. Often times this software comes pre-installed on new systems and the user doesn't even know they have it or what it does. Expect to see the number of valid referrers captured continue to drop as privacy concerns continue to grow among the typical home user.

Personally I feel that the software vendors are largely at fault for not accurately describing what exactly a referring document is. They make the user believe that HTTP Referers is some kind of trail of every site you've ever been to, so of course the user is going to want to check the box that blocks them thinking that you have no right to see where they were before they came to you.

What they don't understand is that it's really more of a "Where did you hear about us?" inquiry from the website. It's not our right to tell a user whether they should or shouldn't block referring documents, but as webmasters it's our responsibility to ask the software developers to do a better job of explaining it in their applications. I'm sure the companies I mentioned have some way to submit comments about their products, I'll see what I can find later.

Lord Majestic




msg:904910
 2:43 pm on Aug 10, 2004 (gmt 0)

On average (over 3 popular websites) I find about 50% of logs have an empty referrer.

I've come across with an interesting "feature" of IT, which is not to set referer for windows opened by JavaScript (Netscape at the time of testing was fine, not sure about Mozilla). Can't be arsed to test now if <a href="url" target=_new> would also reset referrer, but I think it won't.

And why does referer anyway? Proper tracking system would contain right codes as part of URL, so that referrer by itself won't matter - your tracking code is the reference of source that referred click to you.

dataKris




msg:904911
 3:46 pm on Aug 10, 2004 (gmt 0)

Are we talking about the IP field or the referrer field?

The topic of IPs came up as I was trying to say that a large number of AOL users seem to end up showing a very disconnected path (without proper referrals) - that some web master world posts attribute to caching.

digitalv:

it is interesting that such a large number of blocked referrals happen due to these software. I have read other stuff which say that blocking referrals is the next big thing in browsers. That would suck to use webmasters.

datKris

cfx211




msg:904912
 4:04 pm on Aug 10, 2004 (gmt 0)

We see a lot of redirects blank out referrers. For instance one of our partners runs their links through a jsp file that ends up blanking out the referrer.

We also have a couple of other URLs that we own that we redirect to our site and that will blank out the referrer.

If you get a lot of traffic to your site via newsletters and emails, a lot of that traffic will have a null referrer. Sometimes that is because of where the person is viewing the email, but if you have tracking tags on your email that can blank out your referrer.

If someone types in your domain, you get a blank referrer.

If your page is cached, you get a blank referrer.

I feel like this is turning into a Jeff Foxworthy bit.

kapow




msg:904913
 5:35 pm on Aug 10, 2004 (gmt 0)

Thank you Digitalv. Your posts are very illuminating.

Referrers are blocked in Internet Explorer more than any other browser....

Wow! So this means that ROI tracking is very limited in the absence of a tracking url e.g.?referrer=soandso

...as webmasters it's our responsibility to ask the software developers to do a better job of explaining it in their applications.

This is very important!
As you say "...it's really more of a "Where did you hear about us?" inquiry from the website."
I'm sure the software vendors would appreciate the importance of this if the message was make clearly to them by a significant number of webmasters. What would be an appropriate and effective way of sending that message?

Proper tracking system would contain right codes as part of URL

That is only possible for sponsored referrals, the world is much bigger than sponsored! I find about 25%-40% of referrals are sponsored (e.g. adwords, overture etc for which I have a tracking code) and the rest are natural search engine listings (of those I can track!). You can't put codes into naturally spidered pages.

We see a lot of redirects blank out referrers.

As I said, I'm not talking about Affiliate stuff. Affiliate tracking is a different matter and probably requires a thread of its own. I'm talking about tracking ordinary search engine referrals and standard 'Pay Per Click' stuff like Adwords and Overture.

digitalv




msg:904914
 7:08 pm on Aug 10, 2004 (gmt 0)

So this means that ROI tracking is very limited in the absence of a tracking url e.g.?referrer=soandso

Yup :(

What would be an appropriate and effective way of sending that message?

I haven't really thought of that yet, I'm not going to have time to mess with it today but maybe someone else can pick up the ball. I guess the first thing to do would be to get a list together of the applications that offer this feature and take a look at how it's worded in the program. Ultimately some people are going to be pro-active in blocking their referrer just because that's how they are. It's the people who don't really know what they're blocking that need to be educated. I guess there could be one good thing to XP SP2 - with some built-in security perhaps fewer people will use the third party stuff :)

You can't put codes into naturally spidered pages.

I suppose you could do this by detecting the UserAgent and changing the link based on it (ie; insert tracking info into links when useragent = googlebot), but you probably wouldn't want to because it might hurt your positioning down the road. Plus unless your links are served from a database it would be a real pain to change by hand :)

The work around for this is to use two domains. Advertise one of them and configure robots.txt to disallow all search engines. Then have your second site set up without any restrictions in robots.txt, but don't ever advertise it. With a little bit of server configuration you can even make robots.txt dynamic based on the domain name. Since calculating ROI is ultimately more important than links you're not paying for, at least the advertised site will be much more accurate.

kapow




msg:904915
 7:56 pm on Aug 10, 2004 (gmt 0)

So this means that ROI tracking is very limited in the absence of a tracking url e.g.?referrer=soandso

I would be interested in any comparison percentages.
e.g. I make mine about 50% untrackable. Like I said, I'm amazed its so high. Until today I was sure we had a weakness in our tracking system.

So, those of you tracking natural (non sponsored) conversions, what is your tracking success percentage?

DenRomano




msg:904916
 3:12 am on Aug 14, 2004 (gmt 0)

We work our tracking just as you do. I really do not think it is a problem as what I would guess is the same percentage of people that allow the refferal URL as to the ones that do not.

What I am trying to say is if I get 10% of the people coming to my site by say google search for word "widget" for the users I can see the reffer then that same percent I apply to the ones that do not.

This is the same logic I use for people calling in there orders on the toll free line. I can not track them but I apply the same percents from my tracking to those orders as it would seem reasonable that those people would follow the same pattern.

kapow




msg:904917
 11:11 am on Aug 14, 2004 (gmt 0)

What I am trying to say is if I get 10% of the people coming to my site by say google search for word "widget" for the users I can see the reffer then that same percent I apply to the ones that do not.

I agree. However this post is about undetstanding approximately what the percentage or trackable natural referrals is. When I know what I can't track I will stop chasing them and get on with using the intelligence gained from those I can track. Last month I assumed I would be able to track say 80%. Now it seems more like 50%. If thats the reality then ok, I'll stop chasing the impossible.

edit_g




msg:904918
 11:17 am on Aug 14, 2004 (gmt 0)

If thats the reality then ok, I'll stop chasing the impossible.

That's the conclusion I came to also. I've seen everything between 20%-75% blanks across the sites I've worked with. As long as you know you're covering the obvious bases this is something which you either accept or let it drive you crazy.

kapow




msg:904919
 5:50 pm on Sep 22, 2004 (gmt 0)

WOW! I just found out Ad-Aware finds and removes 'tracking cookies'.

Tracking cookies are harmless. If Ad-aware highlights them it makes them look sinister.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved