homepage Welcome to WebmasterWorld Guest from 54.242.18.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

This 44 message thread spans 2 pages: 44 ( [1] 2 > >     
"No referrer" referrals - What are they?
and where are bookmarks grouped in?
vitaplease




msg:894960
 11:22 am on Nov 12, 2002 (gmt 0)

My Webtrends referrers shows the following:

1.http://www.mysite.com 27 %
2.No Referrer 19 %
3.the rest are the search engines and sites linking to me.

Can anyone help me with the following:

Where do the No.1 visitors come from? Bookmarks? People copying the url in the browser window? People clicking on email addresses?

What type of visitor are classed/grouped in No.2 No Referrers?
People some way hiding their identity? Or is it Webtrends not being able to identify the origin of the referral?

many thanks for your input.

 

Sinner_G




msg:894961
 11:26 am on Nov 12, 2002 (gmt 0)

No 1. is just people surfing through your site. The referrer just tells you on what site the last page visited was.

No2. are bookmarks, people just entering your URL in the navbar and such.

vitaplease




msg:894962
 11:35 am on Nov 12, 2002 (gmt 0)

Thanks Sinner,

so yesterday I had 2000 visits from 1650 uniques.

Are you saying 19% were bookmark visits and people spelling/compying the url in the browser address window?

No 1. is just people surfing through your site. The referrer just tells you on what site the last page visited was.

This I do not understand. Are you saying Webtrends sees my own site as a referrer? But then the unique visitors do not add up with the number of visits.

Hannu




msg:894963
 11:49 am on Nov 12, 2002 (gmt 0)

Are you saying 19% were bookmark visits and people spelling/compying the url in the browser address window?

and users clicking on links in e-mails, proxy server filtering of referrals and browsers set to not sending referrals.

...Are you saying Webtrends sees my own site as a referrer? But then the unique visitors do not add up with the number of visits.

Yes it does. And referrals are counted on a per-request basis so actually you can't use the numbers to derive uniques etc.

vitaplease




msg:894964
 4:38 pm on Nov 12, 2002 (gmt 0)

Hannu,

Yes it does. And referrals are counted on a per-request basis so actually you can't use the numbers to derive uniques etc

One day:

2023 visits
from 1664 uniques
of which 1,475 who visited once
of which who visited more than once 189

I would say those uniques who visited more than once resulted in the 2023-1664=359 extra visits (mostly mycompany employees)

referrals from:

www.mysite.com 547 (27.03%)
No Referrer 390 (19.27% )

So basically you are saying the 547 referrals from mysite.com are double counts that should be deducted from my uniques, because they came from mysite.com and are counted as extra seperate visits by a confused Webtrends/Vitaplease?

2_much




msg:894965
 7:59 pm on Nov 12, 2002 (gmt 0)

vitaplease I've wondered this for years and haven't gotten a conclusive answer.

On some sites we have up to 80% no referrals - and there's no way they're bookmarked, coming from links in emails, or typed into the browser.

Can someone shed some light on this?

austtr




msg:894966
 11:42 pm on Nov 12, 2002 (gmt 0)

hannu...

<proxy server filtering of referrals and browsers set to not sending referrals>

More info please.

From the original post, and others, it seems this is not a well understood topic.... can we get some contributions that explain "No Referrals" and what the figure represents?

KISS principle would be appreciated.

jatar_k




msg:894967
 12:06 am on Nov 13, 2002 (gmt 0)

a referer is not always passed.

no referrer could be:

type ins
bookmarks
set as homepage
browser didn't pass referrer or browser set to not pass referrer
proxy server filtering of referrals (from Hannu, don't personally know much about it)
links from email

there are probably more, These are all i could think of off the top of my head

feel free to fill out the list

pageoneresults




msg:894968
 12:13 am on Nov 13, 2002 (gmt 0)

No referrers are also coming from those who have set your page as their default home page in IE. I had a similar issue a couple of years ago. I came to find out that we had about 50 of our 80 employees who had the corporate web site set as their default home page in IE. Go figure!

<edit>I didn't see that jatar_k had set as home page, sorry! But, you'd be surprised what percentage of people do that, especially when it comes to larger corporations.</edit>

jatar_k




msg:894969
 12:21 am on Nov 13, 2002 (gmt 0)

most corporate sites end up with huge no referrer from their own employees. Especially with large numbers of salespeople who use the site as a selling tool.

Skews numbers quite a bit.

pageoneresults




msg:894970
 1:35 am on Nov 13, 2002 (gmt 0)

> Especially with large numbers of salespeople who use the site as a selling tool.

jatar_k, I think that is a very important point to bring up to everyone who is wondering about those no referrers. You described my corporate entities situation. We are mostly a sales force and the site is used to access various online catalogs that are fed into our corporate site. Even today, many still have it set as their default home page. That's okay, I've excluded our IP's from one set of our statistics.

P.S. To answer the question about bookmarks, look under your Page Not Found Errors section. In WebTrends, it looks like this...

Page Not Found (404) Errors

Target URL and Referrer Hits % of 404 Hits

/favicon.ico

ken_b




msg:894971
 1:52 am on Nov 13, 2002 (gmt 0)

Does the favicon.ico show in the 404 list everytime a visitor uses their bookmark to get to a site, or just when they bookmark the site for the first time?

vitaplease




msg:894972
 7:26 am on Nov 13, 2002 (gmt 0)

thanks guys for the imput on "no referrer"

I am actually more confused with the referrals from www.mysite.com and the fact that that does not add up with the uniques.

Because if a referral comes from www.mysite.com, this referral should have already been counted as a referral through "no referrer", a search engine, or a link referral from an external site.

<<added>> the following can add to referrals from your own site as a repeated unique:

from Webtrends help:

Visits - Number of visits to your site. If a visitor is idle longer than the idle-time limit, WebTrends assumes the visit ended. If the visitor continues to browse your site after they reach the idle-time limit, a new visit is counted. The default idle-time limit is thirty minutes. This time limit can be changed by the system administrator

So the normal half hour coffee break, with the site idle in the back-ground, will lead to a new visit from the same unique with www.mysite.com as referral I guess. But it still does not explain the huge number of referrals from www.mysite.com IMO.

Sinner_G




msg:894973
 7:55 am on Nov 13, 2002 (gmt 0)

But it still does not explain the huge number of referrals from www.mysite.com IMO.

It all depends on how the idle-time limit was set. I can't remember now whether it is set on the server or by Webtrends, but it could be that it is set at one minute.

Depending on which Webtrends version you have, you may be able to see the navigation pathes. What I have seen from mine is that (too) many pathes are only one page. And often this page is not the home page but just some page which is probably not bookmarked by anyone and doesn't appear in search engine results. What I learned out of this is that you can't trust the results of those log analysis tools.

Hannu




msg:894974
 9:42 am on Nov 14, 2002 (gmt 0)

vitaplease,
So basically you are saying the 547 referrals from mysite.com are double counts that should be deducted from my uniques, because they came from mysite.com and are counted as extra seperate visits by a confused Webtrends/Vitaplease?

Well, ehh... yes and no ;). From WebTrends support on the issue: "WebTrends will count one referring URL (or no referrer) per visitor session. No more. No less."

Per visitor session is the important part. Users who access the internet with dynamic IP addresses will sometimes get several IP addresses during one session. Each new IP address is counted as a new session (unless you use cookies). So if a user is on mydomain.com/page1.htm and then goes to mydomain.com/page2.htm (using a new IP address) WebTrends will see it as a new session with the referrer mydomain.com/page1.htm.

Check the support note [netiq.com] on the issue.

austtr,
Proxy server filtering of referrals. You can configure (some) proxy servers and firewalls to discard certain information, eg. referring url info from the browser, cookie information etc. I can't find non-tech info on this but try browsing through some of these:

[google.com...]

vitaplease




msg:894975
 10:04 am on Nov 14, 2002 (gmt 0)

Thanks for that explanation and link Hannu,

Also, from that same link:

You have a Busy Web Site

- Each demand placed on a Web server to deliver content is considered a hit (or event). The Web server records each hit as a separate event, and displays those events as separate lines in the log file.
On a busy Web site, the Web server delivers content almost continuously - several times a second. When more than one event takes place at the same time, the Web sever stamps each of those events with the same exact date and time (to the second).

- When WebTrends parses the log file, it has to decide for itself which of those events happened first. WebTrends does that during the synchronization process. The event identified as the first of the visitor session is the one that WebTrends looks at for evidence of referral information.

- If that event contains reference in the referral field to a page from your Web site, then that page (your page) gets credit as the referring URL.

I do not have enough visitors to fall in the above category, but I have several "calculation" Frontpage forms, which get visitors tend to play around with. Maybe that adds some confusion.

cornwall




msg:894976
 10:48 am on Nov 21, 2002 (gmt 0)

Bumping this thread back up

I too have tried for years to get to the bottom of this one. I am giving a talk next week to a client group, and their web stats is bound to come up in the questions. Webtrends themselves do not seem to know the answer - or if they do, they are not telling :(

1. Can anyone definitively answer what happens with caching. For example www.mysite.com is looked at by the client of a major ISP like AOL or anyone else that uses caching. www.mysite.com is cached by the ISP, and spooned out to their next client using their cache.

Their client could have found www.mysite.com on Google, via a referal site, or any other way

My understanding is that there is no record of that visit on my log files at all.

Let us say that the user then clicks an inside page not on the cache. How does that show up on the logs. As a "no referrer" or as "www.mysite.com" or what? For the sake of this example assume that they found me on Google and got the site from their ISPs cache.

My understanding is that there would be no record of Google on my log files, but that "www.mysite.com" would show up as the referrer

2. If the last sentence is correct then does anyone have any ideas on roughly what percentage of visitors come from ISP caches

3. I am still baffled by the thread's conclusions. :)

cornwall




msg:894977
 11:39 am on Nov 21, 2002 (gmt 0)

And...

Having rooted round in AOL's Webmaster info,
[webmaster.aol.com...]
they appear to cache web info for 24 hours and serve it up without re-testing www.mysite.com

The AOL cache will not serve a cached object for more than 24 hours without revalidating it with the origin server even if this value is set to expire the object more than 24 hours into the future.

Last-Modified:

States the time an object was modified. Objects with a ""Last Modified"" header without an ""Expires:"" header will be cached for 20% of the age of the object but not more than 24 hours. The age of the object is determined by subtracting the ""Last-Modified:"" time from the current time:

Maximum Time in Cache before Marked Stale = (Date Served - Last Modified) * 20%

If the requested object is determined to be stale, the cache will send an If Modified Since request to the origin server. The cache will update the object if it has been modified. If no changes have occurred, the object will receive updated Date Served headers and remain fresh until it the age equation determines it to be stale again.

This would appear to mean that (in the case of AOL) they serve up without re-checking up to 24 hours. In other words, no entry in your log file (unless you have set metas to vary this 24 hour rule)

And that after 24 hours all that will appear is an "if modified" request on your log files if another AOL client requests info on your site. If you have not modified your site, then the cached version stays in place for any other users in the next 24 hours and so on.

As I understand that, they only cache any page(s) that the first user requested, after that they will have to request more pages if their further clients request them, from your server. These will show up on your logs.

Still begs the question as to how that shows up on your logs.

Makes me think a big site with lots of pages even if it is updated infrequently will still register visits after periods of time for the above reason

And that small sites, updated rarely (for example an individual hotel site) will probably get a flush of "extra" visits each time they update.

carlosyf




msg:894978
 9:27 am on Nov 22, 2002 (gmt 0)

Hi vitaplease:
There are a lot of doubts about the referers in a log. Any of those doubts can be answered, and other no.
Now you can know for example is it's people clicking on e-mail, or the context of your link on the web-page.
About the problem of the internal referers or no referers we are very much people that is thinking in that case, but there are very few answers. There is a new software (is of my company, I don't want to lie) called VISIT Source (www.visitsource.com), that analyze that context of each site nad link. It's not directed to an analysis of number of visits, and yes about quality of visits.
I would like to know your opinion... if sugarkane allows me ;-)

Brett_Tabke




msg:894979
 9:44 am on Nov 22, 2002 (gmt 0)

Excellent Thread!

There are so many variations on referrer behavior from browsers, that if you are within 20-30% of reality you are doing good.

If your log file will allow you to do it, throw out everything but the first visit for any user. Only use those referrers. That will give you the most accurate account.

Other things that will throw off referrers:

- some browsers will only send the root domain for any site.
- some browsers and proxy servers will repeatidly send an external referrer for EVERY page it visits. If it comes in from Google, and they visit 20 pages, all 20 pages could see that same google referral string sent.
- Most clued in Opera users turn off referrals as a security precaution. Mozilla may have an option to do the same soon. They are arguing about it now.
- I have heard that there is a version of msn IE that will not report an external referral under some security settings (not sure, but the pattern fits).
- Revisits. If a page is reloaded, some browsers will sent that page itself as the referral. hence, the high proportion of www.mysite.com in your logs.
- no cache mania. Most of the dsl, cable, and other high speed modem manufactuers are telling people to turn off caching in their browser. They all have explicit details on their site as one of the setup steps to take. That in turn is skewing referral numbers as even a simple back button can cause a page reload. That referrer will often be the previous page.

It's been my experience that 50 to 75% of insite referrals are not correct. Bookmarks, typed-it-ins, drop down history from address bar, caching, no caching, and reloads have turned insite referral numbers to junk. There are no major log file analyzers that have this fact figured out.

All the "foot step" programs generate extremely false data. Here is one:

Log shows these hits:

- index, contacts, about, superpage, page2, page3

Most footstep programs will show you just that path. When in fact, the only way to get to "page3" is from the superpage. Most have no concept of caching or link structure. Put no weight in foot step programs unless your site is just one page (grin).

chiyo




msg:894980
 10:20 am on Nov 22, 2002 (gmt 0)

Part of this problem is because makers of log analysis programs need to add extra features to justify their gross prices. Problem is that the raw data is very iffy, and its GIGO (garbage-in garbage-out).

We use log data to track over time, (and other modest things like identify broken links), but even that is iffy with ISP's changing cache methods and new products used to access your site.

cornwall




msg:894981
 10:28 am on Nov 22, 2002 (gmt 0)

Brett

Having spent years trying to get to the bottom of what log files really mean, I am not going to let go now till I do :)

Given that we could all (probably) agree that log files are not giving you within 20% to 30% of reality. Perhaps I could try to sumarise what we do know, and ask others to add to the list

1. Comparing monthly log files is relative, rather than absolute. In other words it gives you (broad) trends of traffic looking at the site. If log files are up 10%, then traffic is (probably) up 10%. But they do not give an absolute picture of the real number of visits.
This is presumably true also about the relative importanof referral sites changing over time. If you get twice as many referrals from siteA this month as last month, then in relative terms siteA is delivering twice the traffic

2. Broadly you can ignore browser excentricities other than IE (and to a lesser extent Netscape). Any peculiarities in IE reporting will swamp other browser errors.

3. Bretts point
It's been my experience that 50 to 75% of insite referrals are not correct. Bookmarks, typed-it-ins, drop down history from address bar, caching, no caching, and reloads have turned insite referral numbers to junk. There are no major log file analyzers that have this fact figured out.

4. For a "cachable" site you can be 30% off in you log files
[webmasterworld.com...]
This is a very large figure

5.
no cache mania. Most of the dsl, cable, and other high speed modem manufactuers are telling people to turn off caching in their browser. They all have explicit details on their site as one of the setup steps to take. That in turn is skewing referral numbers as even a simple back button can cause a page reload. That referrer will often be the previous page.

It would be a pity to think that log file analysis is rubbish, but it is difficult to draw any other conclusion that we do not really have any sound idea where web traffic is really coming from.

But it perhaps goes towards explaining why Webtrends and others have not been keen to answer the question!

carlosyf




msg:894982
 10:51 am on Nov 22, 2002 (gmt 0)

If you are all in agreement with the failure of logs... why don't use an extended log?

We know that the log is a file created to control the use of a server, and not to control the users behavior.

However, we can use some software to create a pararel log, merging the web-site structure with the log file. So we can to know reconstruct what the user actually have seen during his browsing.

The case of the referers still can't be solved, but at list we can get the reallity of the movements (or the nearest of that).

Try SCREEN Analyzer (www.screenanalyzer.com) and activate the DataMining files. It will generate a new log file with one register for each page that the user have seen. The software knows the site's structure, and can deduce the movements of the users.

It's recommended to take various coffees. It is a process that can take hours. It's not joke. Launch it today Friday during all the weekend.

Afterwards you can analyze the new log file with your current log analyzer (like webtrends).

rogerd




msg:894983
 2:03 pm on Nov 22, 2002 (gmt 0)

Cornwall, I think your "relative performance" concept is really on the money. There are just two many ways the data can get messed up with the limits of current logging and analysis technology, at least as practiced by most sites.

So what's the answer for more accurate logging? Cookies that are checked on every page? What percent of users have cookies blocked? Session control & tracking?

Fiver




msg:894984
 2:40 pm on Nov 22, 2002 (gmt 0)

Still begs the question as to how that shows up on your logs.

the whole AOL caching thing begs the question, why do we see any consistent AOL referrer strings at all? If they truly did cache anything, wouldn't we have no referrer strings from AOL and lots of 'mysite.com' ones instead?

But we have a fairly consistent number of AOL referrals day to day.

Who uses live web tracking solutions? you know, the kind that puts a graphic on your page and tracks the requests to that graphic as its basis. We try to use both webtrends and a live solution, and even then we are only trying to glean relative information. Or is that what you meant by a 'footprint' analyzer Brett?

I've often had a problem (with more popular sites) of other competitors stealing graphics and games, but popping them right from my clients domain. This skews the numbers beyond recognition, and only for a small time period. So relative month to month is out without a human analyzing the log file themselves and literally looking at pages that are referring out of proportion.

sigh. good to know I'm not the only one scratching my head. bad to know I may never have anything more than relative stats :/

graywolf




msg:894985
 2:59 pm on Nov 22, 2002 (gmt 0)

If you are using webtrends you can filter out your own internal traffic from the reports. I run 4 sets of reports

1 internal traffic only
2 spider/search engine traffic only
3 internal and spider traffic filtered out
4 total traffic

No referrer is the top entry entry in the referer table. Next comes mysite.com, followed by the search engines.

tourist




msg:894986
 4:56 pm on Nov 22, 2002 (gmt 0)

I've got about 8 cents worth to add here, forgive me if I go on...

Regarding no referrers:

Right-clicking in IE & opening in a new window leaves no referrer. (I do this all the time when I don't want to leave a trail, but it can be a bit of a hassle when feeling extra paranoid. If someone knows some settings to prevent referrers from being sent, I'd love to hear it.) I believe that's also the case if you open a new window for your visitor.

2_much awhile ago mentioned up to 80% with no referrers. Do you use JavaScript navigation on your site? I'm not qualified to say all, but at least some JS navigation methods - like the familiar drop-down box - leave no referrer.

One other method of not leaving a referrer? For sites that like to track visitors with redirect scripts on outbound links - like here ;) - drag the link from the page to your Address Bar & release. [google.com...] Webmaster World didn't see you leave & Google didn't see you coming. :) (Works in IE, not sure about others...)

cornwall ponders caching:
Let us say that the user then clicks an inside page not on the cache. How does that show up on the logs. As a "no referrer" or as "www.mysite.com" or what? For the sake of this example assume that they found me on Google and got the site from their ISPs cache.

My understanding is that there would be no record of Google on my log files, but that "www.mysite.com" would show up as the referrer

2. If the last sentence is correct then does anyone have any ideas on roughly what percentage of visitors come from ISP caches

Your understanding has been my experience for the most part. I believe AOL caches the whole page, graphics & all, but that doesn't seem to be the case with some cache servers. I've had a couple of cases where I get a visit from an IP that downloads everything on the page, then a visit minutes later from the same or similar IP with the familiar "Mozilla/3.01 (compatible; )" cache/proxy UA that grabs only the HTML, and then a day or two later the same or similiar IP hits my site and grabs only the graphics with a referrer of mysite.com.

As to what kind of percentages this sort of behaviour represents, I really think it depends on your target audience - or least the type of people who do visit your site. If most visitors come from AOL, then you already know there will be a lot of caching. But if you get "the college crowd," caching of pages seems to be almost nil, in my experience. I figure (many) schools, colleges & universities can't afford the extra cash layout for the necessary hardware.

I'm not familiar with WebTrends - I'm using AWStats while writing my own analyzer because, yes!, the world DOES need another stats program ;) - but if this is how WebTrends works...

- When WebTrends parses the log file, it has to decide for itself which of those events happened first. WebTrends does that during the synchronization process. The event identified as the first of the visitor session is the one that WebTrends looks at for evidence of referral information.

- If that event contains reference in the referral field to a page from your Web site, then that page (your page) gets credit as the referring URL.

...then consider this sample from my logs:

64.229.192.122 - - [18/Nov/2002:00:01:16 -0800] "GET /images/navhome.gif HTTP/1.1" 200 2927 "http://www.mysite.com/index.shtml"
64.229.192.122 - - [18/Nov/2002:00:01:16 -0800] "GET /index.shtml HTTP/1.1" 200 20903 "http://www.google.com/...."

Doesn't happen with every visitor I get, but enough that I don't wonder about it anymore. It sounds like WebTrends would miss the fact the visitor actually came from Google. :(

Receptional




msg:894987
 5:12 pm on Nov 22, 2002 (gmt 0)

I too tackled this issue a long time back and compared many logging systems in some detail.

The truth is there for all to see. Log files do not track visitors. They are a work of fiction dressed up as fact.

Use a Cookie based tracking system. They have their own anomolies but they are closer to the truth than any attempt at measuring using log file data.

tourist




msg:894988
 5:33 pm on Nov 22, 2002 (gmt 0)

Not having ever used a cookie-based system, what quirks does that have?

Found this thread which might explain a thing or two:

[webmasterworld.com...]

rogerd




msg:894989
 6:27 pm on Nov 22, 2002 (gmt 0)

I haven't designed a cookie-based system myself, tourist, but some pitfalls are inevitable. Some users may block cookies, or clear them frequently. More complex, too, is deciding when a new visit begins. Users often don't behave in neat and predictable ways. They may open a page and go away for an hour. Referrer data may be stripped. They may visit your site, leave, and come back in via another link. Without ESP, you'll never know exactly what a user did or intended in every case, so you have to set some basic ground rules and hope for the best. I'd be interested, though, to hear from people who have implemented cookie-based systems. I'm particularly interested in tracking from referrer to purchase to help calculate ROI on ads.

This 44 message thread spans 2 pages: 44 ( [1] 2 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved