WebTrends v6.1c not including UAs on our new Apache server

WebTrends error

[act005]

Hi everyone, surprise, I am a newbie to this industry and this site. If this question is mis-posted, please accept my apology.

We recently moved one of our sites to an Apache server from IIS and now WebTrends isn't capturing the UA field in the log. Troublesome since that's how we filter our bot traffic for reporting. Web Trends is set to auto-detect the logfile type - I tired manually selected Apache Extended and NCSA Combined/Extended, but that didn't work. It captures other data fine, just not the UA - has anyone seen this before?

Here are two entries from the logfile:

[Valid IP Removed] - - [13/Jul/2004:00:42:07 -0500] "GET /robots.txt
HTTP/1.1" 200 121 - nogoop-HttpClient/1.1.3 Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.0) - SSL=-

[Valid IP Removed] - - [13/Jul/2004:15:37:47 -
0500] "GET /MMG/scripts/date.js HTTP/1.1" 200 447
[mnmutualgroup.com...] Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.1) - SSL=-

Any thoughts?

Thanks!
Adam

[cgrantski]

WebTrends (or any log analyzer) has nothing to do with what gets collected in your logs. It's purely your Apache setup. You need to change the LogFormat directive line that's in the httpd.conf file for the site, (you can keep the old directive lines but comment them out if you want). Then restart the server. Exactly how you change it depends on the version of Apache.

[act005]

Thanks for the feedback, but I think my server is logging the UA based on the entries I see. For example in the entry:

[Valid IP Removed] - - [13/Jul/2004:00:42:07 -0500] "GET /robots.txt
HTTP/1.1" 200 121 - nogoop-HttpClient/1.1.3 Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.0) - SSL=-

Isn't the portion:

nogoop-HttpClient/1.1.3 Mozilla/4.0 (compatible;
MSIE 6.0; Windows NT 5.0)

A UA entry?

Thanks!

Adam

[cgrantski]

Aaargh. Yep. I need more sleep I guess.

Well, still trying to be helpful ... I do know that WebTrends needs specific formats in Apache logs. Here's a LogFormat line that they provide users for formatting Apache extended logs:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"\"%{Cookie}i\"" combined

which results in a line that looks like this:

212.959.120.150 - - [31/Jul/1999:16:58:45 -0400] "GET /last5.gif HTTP/1.1" 200 8521 "http://www.une.org/usnews/home.htm" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 95; Toshiba Corporation)" "cookie_id=999999"

As you can see, the WebTrends recommended format puts quotes around a lot of the fields including the UA field, cookie field, etc. Your example lines don't have all these quotes.

Maybe that's the problem.

You could make a tiny log file of just a few lines from your existing logs, add the quotes, and feed it to WebTrends. (be sure to do this odd thing: in the very last line of the log, change the date to the following day. In other words, the last hit should be 24 hours+ later than the next to last hit. This last hit is needed to "close out" the log and it will not be processed.)

[act005]

Thanks, cgrantski, I took that informaiton to the IT deparemnt in my company to prove that there was something wrong with our log files!

Thanks again,

Adam