How to watch and clickthru?

Tools to watch Dynamic URL and clickthru

[emmanuel555]

Hello,

I would like to know the tools you use to watch URL and clickthru with log files

I got Webtrends log analyser but I do not know how to set it up to watch specific URL
(ie: www.mysite.com/description.asp?id=16)

or to know how many customer click on a specific link

Thanks a lot
Emmanuel

[fom2001uk]

I've got the same problem with some of our sites. Basically, WebTrends will NOT list which page the user clicks on (after they've chosen from the dynamic URL) within the PAGES reports (top pages etc). But it will list the exact page (..asp?id=0909etc) in the REFERRERS report.

Other software (like MetTracker) has no problems identifying these pages, so I don't know waht the problem is with WebTrends. I've asked our programmer to look into it.

[confused]

I too have fallen into this limbo world of not knowing where my traffic is going once they get to my site. And since I'm quite new. I'd love to see someone with experience (the advice posted in other forums has been an invaluable resource )post the in's and out's of how to track what and where the traffic goes.

[Hannu]

WebTrends will NOT list which page the user clicks on (after they've chosen from the dynamic URL) within the PAGES reports (top pages etc). But it will list the exact page (..asp?id=0909etc) in the REFERRERS report.

That's because the request fields in the log file are divided into two: "uri-stem" and "uri-query" where the first is the file (page.asp) and the latter the query (?id=0909). The referrer is treated as one string of text.

I got Webtrends log analyser but I do not know how to set it up to watch specific URL (ie: www.mysite.com/description.asp?id=16)

I'm not sure but I don't think Log Analyzer CAN do it. With the Analysis Series Advanced Ed (formerly Enterprise Suite) you have 2 options: "URL Parameter Analysis" and "URL Search and Replace".

With the URL Parameter Ana... you can setup webtrends to report on dynamic URLs (and translate the querys into meaningful strings from a text file).

With the URL Search and R you can replace the ? with eg. a / (forward slash) and thereby force webtrends to treat requests for dynamic pages as unique URLs/pages.

[sagerock]

Hi confused,

Welcome to the boards!

Watching how people flow through a site brings my heart great joy. I especially love tracking them via cookies. I love tracking them right from a key phrase on a particular engine. I love reversing their path through the site once they've done something I wanted them to do.
And I really love watching them over weeks or months come back to my site time and again.

As it stands right now, WebTrends just doesn't work quite that way. In fact, I have yet to get their cookie tracking feature to work. Although, I haven't tried too terribly hard.

So far, the best method I've had for following paths from ANY page is to just import the raw logs into a database or spreadsheet. If I'm using Excel, I just turn on the autofilter and drill down any way I want. It's a bit slow going but at least I can see the logs the way I want.

My second favorite approach is NetTracker.

[amoore]

I have two cutomized tools that I use that work well for me.They both depend on a slightly modified version of the Apache::DBILogger mod_perl module. It logs to a mysql database in real time. I have it add a little more information than it does by default and I have it only log for real pages, not for images.

The first tool is a clickstream report. It depends on cookies, but I suppose that's not necessary. It shows a list of people that came to my site and their prorgress through in a format like this:

1.2.3.10.5811014995858290 <- that's their cookie ID
[example.com...] 2002-03-01 09:17:38
[example.com...] 2002-03-01 09:19:40
[example.com...] 2002-03-01 09:20:56

So that I can see their path through the site.

The other tool is for a much larger view. It is a daily image of the general traffic patters based on Randal Schwartz's article [stonehenge.com] . It shows an oval for each page and then colored lines going between them indicating how frequently the path is taken.

These are pretty customized scripts, but I would be glad to work with anyone who would be interested in trying them out. It pretty much depends on mod_perl apache on linux and a little bit of patience and stuff. Let me know if you're interested.

[sagerock]

Those are very appealing stats, amoore.

Is it Apache dependent? Or could you configure it based on whatever your logs put out?

It would seem that you would just need stats like:
date
time
c-ip (maybe as a backup in case people have cookies turned off)
cs(Cookie)
cs(Referer)
cs-uri-stem

Then as long as you have all of your off-site url's tagged somehow that would pretty much cover it.

Am I on the right track? Or is it more complicated than that?

[amoore]

Really the only thing I have that is apache dependant in this scheme is the module that writes the logs directly to MySQL in real time.

I suppose that if you took your logs an put them in a SQL database that the tools would work alright. well, I think that the graphing one depends on some possliby linux specific graphics libraries, but I'm sure windows has similar tools.

An old version of the script that makes the graphic is here: [cow.mooresystems.com]

and an old version of the one that makes the clickstream report is here: [cow.mooresystems.com]

Feel free to take them and play with them as you want. They borrow heavily frm Randall Schwartz's work. I'm also open to helping you out with tools like this.

[sagerock]

Very interesting.

Thanks for these.

I'll have to spend some time looking these over.

[keyplyr]

>Watching how people flow through a site brings my heart great joy. I especially love tracking them via cookies.

Since IE6 was releashed, it's my opinion that cookies are fast becoming an undependable means of tracking. More and more users are understanding how to prohibit/delete cookies and IE6 makes this feature very easy and accessable. My logs show IE6 to be the second most used browser now.

[sagerock]

I'm seeing about 25-33% of our traffic not using cookies. You are very correct that it isn't fool proof.

Do you use another alternative?

[amoore]

I didn't make my clickstream script track by IP address. I think I should, though for two reasons. First, as you mentined, not everyone accepts cookes, and second, I run several domains on one machine and the cookies are seperated by domain, so tracking users across domains more difficult wth cookes.

All this discussion about tracking users and statistics and logs and stuff made me write a mod_perl module to generate realtime reports out of my webserver log database. I put it up an example at [gotany.com] . I may continue development on it and make it more usable if I get a hankering or someone expresses interest. It's kind of neat to watch. I will probably put it behind a password or something after a few days because I don't like leaving this stuff exposed, but I thought you guys may find it interesting.

[sagerock]

That's a pretty slick little script amoore.

I like it because I often just want to quickly see where people are coming from. I guess it's the voyer in me :)