Cookies, Domino and WebTrends Log Analyzer 8.1

[hotsnot]

Hi all,

I'm running WebTrends Log Analyzer Advanced Edition Version 8.1.
I know ... I'm well overdue an upgrade.
Anyway I'm having trouble getting it to filter based on cookie values from my Lotus Domino logs.
Here's a sample line from the log:
xx.xx.xx.xx www.domain.com - [30/Apr/2006:00:00:01 +0000] "GET /content/portal.nsf HTTP/1.1" 200 16409 "http://www.domain.com/content/cms.nsf" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; Hotbar 4.6.1)" 453 "tsid=0; UserRegion=fr" "d:/lotus/domino/data/content/portal.nsf"
I'm trying to filter hits based on the value of UserRegion and I'm getting no hits at all even though there are several thousand per day.
Anyone had similar problems or recognise this and know a fix?

-Daragh.

[gregbo]

What commands are you using to do the filtering? Please provide examples from config files, etc.

[hotsnot]

Hi,

Thanks for your reply.

>> What commands are you using to do the filtering? Please provide examples from config files, etc.

I'm not using config files (I never have ... maybe I should be?).
I'm using the filter manager in the WebTrends user interface.
Here's a shot of the filter as I've set it up:
<snip>
It's looking for hits with cookies where the value of UserRegion is set to na.
This happens thousands of times in the logs yet I'm seeing zero hits in the report.
As you can see from the shot, I'm not combining the cookie filter with anything else that could be screwing things up.
Let me know if you need more info and thanks again for the response.

-Daragh.

[edited by: Receptional at 8:44 am (utc) on May 13, 2006]
[edit reason] Sorry - no link please [/edit]

[hotsnot]

Hi,

I've tracked the cause of the problem down but still haven't found a solution.
The problem is related to the fact that Domino is logging the processing time for the request and it must be logging it in a non standard place. In the example above, it's the 453 figure just before the cookie string. I've manually removed that value from a few of the log entries and lo and behold those hits are appearing in the report!

So my new question is:

Is there anywhere in WebTrends that I can configure a custom log format and have it added to the Log File Format dropdown in the Log Data Sources definition dialog?
Or can I redefine how it interprets the Domino log file format somewhere?
Or is there any other way to convince Webtrends to look in the right place for the cookie value?
Unfortunately it appears that it's not possible to change the format on the Domino side.

Any ideas here would be greatly appreciated.

-Daragh.

[gregbo]

Is it possible for you (or someone) to filter out those processing time entries and then resubmit the logs to WebTrends?

[hotsnot]

How would you do that?
There's tens of thousands of hits per day so it couldn't be done manually.
Are you saying that WebTrends can't be configured to read the logs in the format they're saved?

-Daragh.

[cgrantski]

Your version of the program might predate Domino logs. I know I've seen Domino on a list of log configuration choices in old versions of WebTrends, but maybe not that old. If you don't want to update, you can write a preprocessing script that removes that field. It's not that hard since the delimiters in Domino logs are pretty straightforward.

[hotsnot]

Actually Domino is an option in the list, but I guess the format that Domino uses has changed since the version of WT I'm using was released so it doesn't work any more. We have an up to date version of Domino, but an old version of WT.
So are you saying that user defined log formats can't be set up in WT?
What about in the current version of WT?
So the preprocessing thing you mentioned ... I haven't used that before ... are there examples of it online that you could point me to and is it pretty reliable?

Thanks for all you help so far!

-Daragh.

[gregbo]

I was thinking that if you know Perl, for example, you could write a script that replaces the numeric string that represents the processing time (and the space following it) with the empty string. You then feed your logs to the script and pass the output to WebTrends.

[hotsnot]

Couple of things:
1. I don't know Perl. Though I have a Find / Replace app that works with regular expressions.
Anyone figure out a regular expression to match the Processing Time field in the log sample above?
2. If possible I'd like to keep in the Processing Time field as we may want to report on that at some stage and it's possible that a new version of WT will pick it up correctly.

Sorry to keep harping back to this but I'd really like to get a solution within WT if possible:

1. Can user defined log formats be set up in WT LA AE 8.1?
2. What about in the current version of WT?
3. Does the current version of WT recognise the Processing Time and Cookie fields correctly from Domino logs?

My preference here would be to upgrade WT and, if I can say that upgrading will definitely solve our problem, I'll have a much better chance of getting the budget.

Thanks!

-Daragh.

[gregbo]

I'm not sure what the exact syntax of Domino logs is, but I'm assuming that the last two fields are quoted strings.
The following script (not fully tested), when it encounters a numeric string followed by a quote (where you don't need to recognize anything specific within the quotes), eliminates the numeric string and prints out everything else until the end of the line:

while (<>) {
 if (/^(.*) (\d+) (\".*)$/) {
 print "$1 $3\n";
 }
}
 

I'm sure there are a lot of corner cases that need to be tested, but this will give you a start, at least. IMO, your best bet is to try to get the people who produce the logs to generate them in a standard format that WT recognizes.

[hotsnot]

Hi,

Thanks for the Regex!
Unfortunately, the output of the Domino logs can't be done any differently, so I'm stuck with those.

Any thoughts on my other questions:

1. Can user defined log formats be set up in WT LA AE 8.1?
2. What about in the current version of WT?
3. Does the current version of WT recognise the Processing Time and Cookie fields correctly from Domino logs?

-Daragh.

[gregbo]

Unfortunately I don't know the answer to your other questions.

I'm curious why the log format can't be changed. Who is responsible for generating the logs?

[hotsnot]

>> Unfortunately I don't know the answer to your other questions.

Anyone else on here know these answers or point me in the right direction?

>> I'm curious why the log format can't be changed. Who is responsible for generating the logs?

The Domino server generates the logs and there's nowhere that we're aware of to change the format.
Do you know if this i spossible?

-Daragh.

[cgrantski]

I found these instructions for (supposedly) getting Domino to output in standard extended common log format:
*************************************
Set up your Domino server to log to text files. Open the server document in your Public Name and Address Book/Domino Directory and go to the "Internet Protocols - HTTP" tab, and set the following fields:

Access Log Format = "Extended Common"
Time Format = "LocalTime"
Log File Duration = "Daily"
Max Log Entry Length = "10 kb" (release 6 and higher)
Max size of access log = "0" (release 6 and higher)
Enable logging to log files = "Enabled"
Enable logging to Domlog.nsf = "Disabled"
Directory for log files = "d:\domino\weblogs"
Save and close the server document, and restart HTTP on the server.
*****************************************

If this does produce extended common logs, you should have no trouble getting even ancient versions of WebTrends to work with them.

I suggest in WebTrends that you set the log file format choice to "Autoconfigure" if you haven't already, rather than Domino.

You cannot change WebTrends' built-in configurations for log file reading, or make new ones. You also cannot use WebTrends' Search & Replace function to get what you want because that feature only works on the URL field and the query field.

If the above steps don't work, your only option, even with new versions of WebTrends, is to preprocess the logs to remove the problem field(s). If you can't find somebody to write a Perl script, try the program called LogParser that's available on the Microsoft web site. It will definitely do the preprocessing you want and it's not too hard to learn to use. Preprocessing like this is commonly done in web analytics.

As for your question about whether the current version of WT will process these logs, I suggest you find somebody with the current version and give them a log.

[hotsnot]

Hi cgrantski,

Thanks for the suggestions (and good talking to you again!).

On the Domino setup, I actually have the logging set up as you've suggested.
The problem is that the only options there are for Common and Extended Common - there's no way to configure the exact set of fields that are logged.
Unfortunately Domino's Extended Common format doesn't match the one WebTrends is looking for; the problem being the Processing Time field. I've tried the Autoconfigure Format option as well as the Domino Format option (and several others) in WT and nothing works. I'm wondering if the latest WT is aware of the latest Domino format.

>> You cannot change WebTrends' built-in configurations for log file reading, or make new ones.

Bummer! ... Seems like this would be very useful functionality.

>> You also cannot use WebTrends' Search & Replace function to get what you want because that feature only works on the URL field and the query field.

Bummer again.

>> try the program called LogParser that's available on the Microsoft web site.

I'll check this out.

Thanks to all for the help!

-Daragh.

[gregbo]

I suspect part of the problem (on WT's part) may be that the last two fields in the extended common log format (according to some web sites I checked) are supposed to be the referrer and user agent, not cookies and the pathname that were in the sample log entries given. WT may be confused by those extra fields. What exactly does WT say the extended common log is supposed to contain?

[hotsnot]

Hi gregbo,

Actually I tracked the problem to the fact that Domino is logging the Processing Time for each request. This is being logged at the field position just before the Cookie field (see message #4 above).

cgrantski,

I've downloaded LogParser and have been learning how to use it for the last about 16 hours straight. It's very cool and I had no idea it existed before!
I've been able to set it up to filter out the Processing Time field and also do a whole bunch of other cool stuff that I had wished was possible. WebTrends is now picking up the data as expected. Thanks for that pointer!

-Daragh.

[cgrantski]

Hurray for you actually following up and learning the tool. It is one of the great bargains (free) in the web analytics world.