Better log analysis for Linux

Webalizer just doesn't cut it.

[Nick_W]

Hi all, looking through my webalizer stats I'm finally realizing how woefully inadequate they are.

Presuming I can download the raw log files to my Linux box: What's my best option?

I'm noticing inexcusable descrepencys between my raw files and my stats. My site gets a hit a few times an hour for one phrase and it does not show in my webalizer stats at all!

Apart from that, I just need more informaion ;)

Thanks

Nick

[digitalghost]

Sawmill is pretty decent, but I prefer FastStats. Sawmill has the advantage of being free.

[Nick_W]

Right! Well it seems I have Sawmill so:

$> man sawmill

it is then ;)

Cheers...

Nick

[dingman]

'analog' is the standard against which all others seem to compare themselves, so you might want to try that. It's extremely configurable and I am only beginning to think at all about website statistics, so I have little experience to base any analysis on, but there you go. There's also something called 'awstats' that claims to be better and easier to use than webalizer.

[Nick_W]

....er

Sawmill is/was a windows manager (now called sawfish) no man entry and doesn't appear to be the right tool anyway...what gives?

Nick

[dingman]

I thought Sawmill was a window manager... in fact, I'm pretty sure it's the one I'm using :)

[Mark_A]

Nick_W - might you be able to set up webaliser a little better?

Webaliser needs you to tell it the preceeding code before the search term for each search engine - in its configuration - for it to correctly identify the key word from the search / referral string. If you dont check that for every engine then you may not get correct key terms results.

After all its not as if all the engines keep the same referrer information over time. Compare to your raw logs and adjust accordingly.

imho for free you dont get much better value for money than webaliser :-)

Example from a standard config ... (may not be up to date)

#MangleAgents 0

# The SearchEngine keywords allow specification of search engines and
# their query strings on the URL. These are used to locate and report
# what search strings are used to find your site. The first word is
# a substring to match in the referrer field that identifies the search
# engine, and the second is the URL variable used by that search engine
# to define it's search terms.

SearchEngineyahoo.comp=
SearchEnginealtavista.comq=
SearchEnginegoogle.comq=
SearchEngineeureka.comq=
SearchEnginelycos.comquery=
SearchEnginehotbot.comMT=
SearchEnginemsn.comMT=
SearchEngineinfoseek.comqt=
SearchEnginewebcrawlersearchText=
SearchEngineexcitesearch=
SearchEnginenetscape.comsearch=
SearchEnginemamma.comquery=
SearchEnginealltheweb.comquery=
SearchEnginenorthernlight.com qr=

Hope this helps

[digitalghost]

There is a Sawmill for Linux version folks. A search will find it. Some of my clients have access to raw logs, others do not so I have to use whatever I can to get the best stats I can get. Out of all of them, Webalizer seems to be the weakest.

[Mark_A]

Hi digitalghost I see you are pretty sold on sawmill I have never seen that one .. I think if Nick_W confirms that his webaliser is configured correctly he might like it rather better (again as I mentioned cost benefit does not get much better) I find it fine as a basic quick check on activity - there are always logs and offline dbases for detailed work.

fwiw I hated analog the one time I had to look at it.

The online one I like the best is urchin .. allows quick easy deep drill down into individual search engine referrals etc etc all online .. the only negative I could see was that the version I get to play with does not provide paper reports which some larger clients would rather have than online access ..

[Nick_W]

Hi,

Thanks everyone. Can't confirm if Webalizer is configed right as I'm just not up to huge edit jobs tonight ;)

But, I can confirm that Sawmill is absolutely great! just gotta check out what it'll cost me after my 30 days is up ;)

Nick

[4eyes]

We use webalyzer on all our accounts.

It is excellent at what it does once you realise its limitation.

If you modify the config files wou can get a lot more out of it than the default settings - which frankly are poor.

However the big 'down' is that it won't show key phrases by engine - ie you get key phrases and you get search engines - but you can't see which came from where.

For this I use 'Relax' - its a free perl script which I run locally on downloaded files. It specialises in Search Engine referral analysis and is better than anything else I have tried - very detailed and covers loads of engines.

[scotty]

I also had a look at Sawmil, and played through its samples. Very nice product, but too bad it is not free.

Personally I use AWStats [awstats.sourceforge.net] which is available at Source Forge. It is driven by Perl so it is a bit slower than Webalizer. However it provides a lot more information regarding to bots, search engine referrals, browser/OS identification, etc. Most importantly, it is in active development, whereas webalizer has not had a serious update for ages...

[Nick_W]

Digital ghost showed me this link:
[sawmill.net ]

Where you can get it free! ;)

Nick

[martin]

I've used webalizer, AWStats, http-analyze, Analog, and a few others.

The most correct statistics are produced by Analog but they are not as fancy as the others. Get Report Magic if you want fancy reports.