Forum Moderators: DixonJones
On average (over 3 popular websites) I find about 50% of logs have an empty referrer.
How many of yours are empty? Any solutions?
Note: we don't do any affiliate stuff its just natural and sponsored search engine visitors we're tracking.
I've always maintained that if the source of traffic is hidden then what lies beneath it, but as some of it converts to sales I can't be a total sceptic and assume it is all bad.
I think the PPC providers owe it to advertisers to ensure tracking abck to source, and encourage them to stop paying affiliates that deliver traffic with no source.
Here are some previous posts that have discussed the same issue. Will post more later today:
[webmasterworld.com...]
[webmasterworld.com...]
dataK
most of my hits like that are from spiders
I'm not talking about just traffic, I'm talking about conversion tracking ie those people who bought something sometime (minutes or weeks) after visiting the site. We capture the referrer on first visit and then make checks when they buy (no need to go into all the details).
The problem is the initial referrer logs are empty for a lot purchasers! ie no referring website. As far as I can tell that means they didn't come from anywhere e.g. they typed the domain into their browser (which is possible as the sites are popular).
Or am I missing something?
For blank referrer info, read message 38 of this thread:
[webmasterworld.com...]
My biggest issue is with the amount of sales I have with no external referral. About 20% of the sales seem to always start at the addToCart page on my site. We kept theorizing it maybe due to bookmarks until we started doing PPC. Some of the links are so unique and so fresh, I can manually go through my logs and pick the IPs that visited those pages. There is no way those could have been bookmarked.
After I read through some postings, I realized what maybe happening. My product pages were getting cached and when the buyers attempted to add to cart or checkout, they were hitting my servers.. Further, a very random check of the whois info of the sale IPs show a major presence of AOL powered IPs.
I can't think of a single thing to do when my pages get cached.. I am almost sure that if I solve this, I would boost my accuracy by atleast 20%.
Any heads up would be appreciated.
the referrer string is empty ie no trackable referrer.
Many people either block the referring document through software (like Norton Internet Security, McAfee, a couple of Firefox plugins, etc.) and many others are surfing through anonymous proxies that do this automatically. Basing your tracking on the referring document is unreliable. I suggest you use a Tracking URL for your advertising sources. Rather than have your advertisements link to www.example.com have them link to www.example.com/?track=source
Then all you have to do is look in your logs for how many hits there were to /?track=whatever and you'll know how many clicks were delivered - you can ignore the referrer completely.
I can't think of a single thing to do when my pages get cached.. I am almost sure that if I solve this, I would boost my accuracy by atleast 20%
Subscribe to a hosted service (iwebtrack, webtrends live, etc.) and put the Javascript code in your pages. Even if the page is cached, the user will still be tracked because the Javascript will register a page view on the stat provider's servers every time. I use both the raw log files and a hosted service and between the two I get a much better idea of how many visitors there are and where they are coming from. But definitely use a tracking URL instead of the referrer when it comes to determining where your visitors came from.
I do appreciate your post. But, here are the issues.
I have my own javascript based logs and also look at the raw logs once in a well. Both show me the same situation - the log captures the IP only starting from the https pages or the dynamic addToCart page. Also, how could I explain the major presence of AOL IPs - a coincidence?
Thanks anyway.
*could* be a problem with your cookie
use a Tracking URL for your advertising sources.
Are we talking about the IP field or the referrer field?
That is interesting! Some notes on it:
Re. 'Some browsers won't send a referral string'
I know there are lots of different browsers but like it or not I find 85%+ of visitors use IE.
Re. '...proxy server or other filtering agent...'
Surely thats a small minority. I'm talking about ordinary sites selling ordinary stuff to ordinary folk.
I find 85%+ of visitors use IE.
Referrers are blocked in Internet Explorer more than any other browser - your own logs should prove that. As I mentioned before, programs like Norton Internet Security, some versions of McAfee, and lots of privacy/anti-spyware/adware software does this automatically. Often times this software comes pre-installed on new systems and the user doesn't even know they have it or what it does. Expect to see the number of valid referrers captured continue to drop as privacy concerns continue to grow among the typical home user.
Personally I feel that the software vendors are largely at fault for not accurately describing what exactly a referring document is. They make the user believe that HTTP Referers is some kind of trail of every site you've ever been to, so of course the user is going to want to check the box that blocks them thinking that you have no right to see where they were before they came to you.
What they don't understand is that it's really more of a "Where did you hear about us?" inquiry from the website. It's not our right to tell a user whether they should or shouldn't block referring documents, but as webmasters it's our responsibility to ask the software developers to do a better job of explaining it in their applications. I'm sure the companies I mentioned have some way to submit comments about their products, I'll see what I can find later.
On average (over 3 popular websites) I find about 50% of logs have an empty referrer.
I've come across with an interesting "feature" of IT, which is not to set referer for windows opened by JavaScript (Netscape at the time of testing was fine, not sure about Mozilla). Can't be arsed to test now if <a href="url" target=_new> would also reset referrer, but I think it won't.
And why does referer anyway? Proper tracking system would contain right codes as part of URL, so that referrer by itself won't matter - your tracking code is the reference of source that referred click to you.
Are we talking about the IP field or the referrer field?
The topic of IPs came up as I was trying to say that a large number of AOL users seem to end up showing a very disconnected path (without proper referrals) - that some web master world posts attribute to caching.
digitalv:
it is interesting that such a large number of blocked referrals happen due to these software. I have read other stuff which say that blocking referrals is the next big thing in browsers. That would suck to use webmasters.
datKris
We also have a couple of other URLs that we own that we redirect to our site and that will blank out the referrer.
If you get a lot of traffic to your site via newsletters and emails, a lot of that traffic will have a null referrer. Sometimes that is because of where the person is viewing the email, but if you have tracking tags on your email that can blank out your referrer.
If someone types in your domain, you get a blank referrer.
If your page is cached, you get a blank referrer.
I feel like this is turning into a Jeff Foxworthy bit.
Referrers are blocked in Internet Explorer more than any other browser....
...as webmasters it's our responsibility to ask the software developers to do a better job of explaining it in their applications.
Proper tracking system would contain right codes as part of URL
We see a lot of redirects blank out referrers.
So this means that ROI tracking is very limited in the absence of a tracking url e.g.?referrer=soandso
Yup :(
What would be an appropriate and effective way of sending that message?
I haven't really thought of that yet, I'm not going to have time to mess with it today but maybe someone else can pick up the ball. I guess the first thing to do would be to get a list together of the applications that offer this feature and take a look at how it's worded in the program. Ultimately some people are going to be pro-active in blocking their referrer just because that's how they are. It's the people who don't really know what they're blocking that need to be educated. I guess there could be one good thing to XP SP2 - with some built-in security perhaps fewer people will use the third party stuff :)
You can't put codes into naturally spidered pages.
I suppose you could do this by detecting the UserAgent and changing the link based on it (ie; insert tracking info into links when useragent = googlebot), but you probably wouldn't want to because it might hurt your positioning down the road. Plus unless your links are served from a database it would be a real pain to change by hand :)
The work around for this is to use two domains. Advertise one of them and configure robots.txt to disallow all search engines. Then have your second site set up without any restrictions in robots.txt, but don't ever advertise it. With a little bit of server configuration you can even make robots.txt dynamic based on the domain name. Since calculating ROI is ultimately more important than links you're not paying for, at least the advertised site will be much more accurate.
So this means that ROI tracking is very limited in the absence of a tracking url e.g.?referrer=soandso
I would be interested in any comparison percentages.
e.g. I make mine about 50% untrackable. Like I said, I'm amazed its so high. Until today I was sure we had a weakness in our tracking system.
So, those of you tracking natural (non sponsored) conversions, what is your tracking success percentage?
What I am trying to say is if I get 10% of the people coming to my site by say google search for word "widget" for the users I can see the reffer then that same percent I apply to the ones that do not.
This is the same logic I use for people calling in there orders on the toll free line. I can not track them but I apply the same percents from my tracking to those orders as it would seem reasonable that those people would follow the same pattern.
What I am trying to say is if I get 10% of the people coming to my site by say google search for word "widget" for the users I can see the reffer then that same percent I apply to the ones that do not.
If thats the reality then ok, I'll stop chasing the impossible.
That's the conclusion I came to also. I've seen everything between 20%-75% blanks across the sites I've worked with. As long as you know you're covering the obvious bases this is something which you either accept or let it drive you crazy.
Tracking cookies are harmless. If Ad-aware highlights them it makes them look sinister.
