Can anyone recommend any other ways of interpreting the data from log files...
ADAM
To back up Webtrends we run our own analysis routines looking for spiders and better engine analysis. In particular, we like to know which pages are found in which engines for which keywords - something I've never managed to get WT to do.
WT also manages occasionally to report referrers that don't exist in the log files. I've given up trying to work that one out. ;) I've cross-checked reports regularly but I've never found serious problems with its reliability, just a few glitches here and there.
You have to play with the config files to get all of the info you need, but you should be able to isolate referrers well enough to separate all of the UK engines from their US counterparts. Although the output is not as pretty as WT, you can load it into a spreadsheet or database and continue processing from there.
The search engine section is limited, but the "referring domains" and "referring pages" reports are much more detailed. When I export the report as a CSV file and use a spreadsheet for further analysis I can really zero in on a lot of valuable information. The referrals reports retain the URL for each referring page, so all the regional search engines pop right out.
Unfortunately the "referring domains" report doesn't pick up search terms, since it was written to pick all kinds of links, and not just search engines. But by knowing what to look for, I can then search the raw files as I need to.
FastStats has some good user defined filters, such as include OR exclude by IP OR domain -- with wildcards. So I can often zero in on what I need to know.
It's a good product for the money, and it really is FAST. But there are still frustrating limitations and potentially misleading numbers -- especially with the hits from proxy servers like AOLSearch or Deutsche Telekom, or the image requests coming from Google's cached page.
I've been wondering if WebTrends plus FastStats might not be a good way to go -- I could compare two different snapshots of the same log file. But WT costs a lot more and I'm on a low budget at present.
