| 7:07 pm on Nov 22, 2002 (gmt 0)|
Now you would think that WebTrends, and others of the same ilk, would monitor this forum out of commercial interests!
The fact that there is no input means either they don't, or if they do, they are so ashamed at the drift of the discussion that they are not contributing :(
The only thing I seem to have really established is that Log files mean something when viewed relatively (month on month). And they can give you a rough idea of the relative importance of where your traffic is coming from.
Given I can get relative traffic movements merely from raw log files sizes on a weekly basis, almost wonder why I go to the trouble to analyse them :(
On top of that I have to stand up at a clients conference next week and explain to a group of non technical hotel keepers just what their stats mean. You can just see the first question coming to me from the fromnt row
"Can you please explain why most of our web traffic comes from no referrer and from our own site". An interesting question to answer in a few minutes to non web people without appearing either a charlatan or an ignoramus (or possibly both)
Still in retospect the thread has been very useful and has clarified my knowledge of Log File analysis
| 7:38 pm on Nov 22, 2002 (gmt 0)|
Cornwall, when presenting data to a client I'd recommend stripping out the no-referrer data and self-referrers. It adds no useful data to the discussion, and just confuses things. I see nothing shady about this, but you could always title the report "Identifiable External Referrers" or similar. Some logging programs will let you suppress this data before the report is even generated.
| 8:00 pm on Nov 22, 2002 (gmt 0)|
Thanks, I'll try that and see what the "rump" looks like!
| 10:02 pm on Nov 22, 2002 (gmt 0)|
|I'd recommend stripping out the no-referrer data |
Don't forget that no referrer can have a positive aspect also: Users have typed in the URL directly. This could be important to eg. a large brand. So instead of taking it out, I would emphasize the positive aspect...
Also, let's not forget that despite all the problems connected with log file analysis as discussed in this thread, we can still meassure a 100 times more than all the TV- and "off-line guys"... ;)
| 10:41 pm on Nov 22, 2002 (gmt 0)|
|Also, let's not forget that despite all the problems connected with log file analysis as discussed in this thread, we can still meassure a 100 times more than all the TV- and "off-line guys"... |
Very true. Brings to mind the old quote
|"Half the money I spend on advertising is wasted. The trouble is I don't know which half" |
| 11:35 pm on Nov 22, 2002 (gmt 0)|
have you tried extracting your log files and trying to use another software pakage to try and read them better?
| 8:33 am on Nov 23, 2002 (gmt 0)|
|have you tried extracting your log files and trying to use another software pakage to try and read them better? |
Forgive me if I am wrong, but I thought we have established here that no matter which log file analyser you used, you could never establish how many visits had been made to your site :(
Caching, re-loading, ISP eccenticities, browser eccentricities and so on, mean that only a percentage of users ever get recorded on your log files. No matter what you use to read the raw log files, if the info is not there, then you cannot analyse it
(I am off now for a few weeks so if I don't reply to any further posts, forgive me. Really interesting thread!)
| 8:57 am on Nov 23, 2002 (gmt 0)|
Summary from this thread:
* Some browsers won't send a referral string:
-- An unsupported or optional feature. Many browsers have options to turn off referral string generation as a security precaution.
-- When "open this link in a new window" is selected in the browser.
-- if security settings are high.
-- if using a proxy server or other filtering agent.
-- if it is a secure page. Some browsers won't send an external referral string as a security precaution.
* Most browsers won't send a referrer when:
-- a bookmark is selected,
-- the url is typed into the address bar,
-- some other program launches the url (such as email or news).
-- if the user has your page set as their homepage
-- from some local file based link (viewing a disk based html file).
-- from page links dragged to the address bar on browsers that support it.
* Some browsers will generate false referrers when:
-- they are located on an isp with a proxy cache. This could mean people from that isp are being vastly under counted.
-- if they are on a isp that has a proxy cacher and that proxy cacher is set to rerequest pages with unique referrers. The rest of the pages that have identicle referrers to the previous cached document, will come out of the isp cache. They could view 50 pages and you only show 10 in your logs with unique referrers.
-- browser caching. Some browsers will repeatidly send the initial external referrer to every page it visits on a site.
-- cache validation. Some browsers send a referrer with "If modified since" requests or even simple "head" checks.
-- reloads. If the page is reloaded, referrer may be that very page or the original referrer to that page.
* Dynamic Content
-- Many sites are mixes of dynamic and static content. Most dynamic documents are not cacheable (cgi based urls). Thus, they can skew referral data dramatically.
-- No longer valid to use for detecting bookmarking of your site. Mozilla, Netscape, and Opera can request the file at every visit to every page depending on settings. That completely skews the data now.
| 10:42 am on Nov 23, 2002 (gmt 0)|
Excellent summary Brett
Our server logs are a bit dodgy & a bit misleading, but they are all we've got , and I love 'em:)
Thanks to all who posted
| 12:49 pm on Nov 23, 2002 (gmt 0)|
(Oops, is this thread considered "over and done with" now?)
It looks as though, if you are able to configure your Web server, you can reduce the amount of caching from proxy servers. Specifically, you should be able to tell AOL not to cache any page.
In looking over the AOL caching info at suggested by cornwall at [webmaster.aol.com ], I came across this:
|The following HTTP headers are used by AOL's cache to determine an object's cacheability. Web Servers can be configured to return the appropriate HTTP headers for the caching behavior you determine to be appropriate. This functionality can be used by a webmaster who may prefer to cache only pieces of their web site. The specific information for configuring a web server is server-dependent... |
This object may not be stored in any cache, even the requestor's browser cache.
Now the Cache-Control HTTP header may not be respected by any other servers, but if AOL truly abides by this spec, implementing it should result in a (somewhat) more accurate picture of file demand for your site from AOL.
Is this correct?
| 1:15 pm on Nov 23, 2002 (gmt 0)|
Here's a resource I return to when I catch myself leaping to conclusions about referrer data: Jeffrey Goldberg's 1995 treatise "Why web usage statistics are (worse than) meaningless" ([goldmark.org ]).
He admits, "On re-reading my original, I see that this document is a bit hyperbolic. So be it. It is after all an acknowledged rant." Nonetheless, it's a realistic appraisal of the difficulty of trying to infer actual use of a Web site (as opposed to inferring actual use of a Web server) using server logs.
| 4:24 pm on Nov 25, 2002 (gmt 0)|
Wow, a lot of input, thanks guys.
I guess I was internally patting myself a little too much on the back when regarding unique visitors these last months.
Logs are fine for indications, trends, errors, keyphrases, bots, but not for high-tech refinement yet I suppose.
Brett, has Webtrends or any other logging company ever contributed to these forums? Would it be an idea to invite them?
| 5:51 pm on Nov 25, 2002 (gmt 0)|
My web stats program (Urchin) shows the typical referral pattern, No referrals, google, yahoo, etc. , but does not show page referrals from my own website. With this, I have always assumed that internal webpage referals are included in the "No referrer" data. Is this a safe assumption?
| 12:05 pm on Nov 28, 2002 (gmt 0)|
We have not invited any of the software guys in Vita because it would lead to problems - flames and competitors taking shots and opportunities. We try to stay away from software discussions whenever possible.
| This 44 message thread spans 2 pages: < < 44 ( 1  ) |