Forum Moderators: DixonJones
Last week Mach5 released version 3 of their FastStats log analyzers. Are you wary of "evergreening"? That's the practice in many software companies of making changes which are not really much of an upgrade. But they give the software a whole new version number anyway, just to jack up sales.
Well, that's absolutely NOT what Mach5 did with FastStats. This new version (I'm running "Gold") is a very ambitious step beyond anything this reasonably priced analysis package ever offered before. And, yes it is reasonable...less than half of it's famous rival, WebTrends.
Let me mention up front that I have nothing to do with this company. I just pay THEM money. But I thought you'd all appreciate a report, since we talk about the program a good bit here. So I asked permission to post this review.
WHAT'S GOOD?
SPEED
For those that are unfamiliar, FS is arguably one of the fastest log crunchers around. With DNS lookup turned off and crunching the files on a local machine, I can rip through 250MB of standard Apache logs in about one minute. And that's on a not-too beefy notebook (750mhz, 128ram). Beefier machines can clock upwards of 600 MB per minute, I've heard.
Version 3 seems to be at least as fast as Version 2 was - and that's saying something because it does so much more.
REPORTS
This is where the real goodies are. Standard reporting includes a new Site Stickiness section for an overall total of 29 reports or so. The new stickiness reports include:
Total Visitor Stay Length
Average Visitor Stay Length
Average Page Views Per Session
Site Entry and Exit Pages
Single Access (one hit wonders)
Average Stay Length (time and total pages)
----by Referring Domain
----by Referrer
----by Search Engine
----by KW Phrase
----by Entry Page
This is information that I used to tweeze out with Grep and Excel in a rather painstaking procedure when I used version 2 -- and now it's all automated for me. Another valuable new report is an analysis of query string variables. And did you catch Site Stickiness by KW phrase and by Search Engine or referrer? That's some great information to have at your fingertips.
This is all very practical stuff with high business relevance - not just added fluff to justify a new version number. My hat is off to Mach5 for this bit of integrity
OTHER GOODIES
Multi-threaded DNS look-up (up to 64 threads at once)
Active URLs in the reports
Many customizable filtering options
Improved Browser stats (this was badly needed)
Individual file tracking within each report configuration
Refering URLS are related directly to their "GET" page
Teoma is added to the standard Search Engine list
SO, WHAT'S THE DOWNSIDE?
Well, as I said, it is ambitious. Like anything ambitious, it has some bumps that could be smoothed out.
WEAK DOCUMENTATION
With so many reports and filters, it would help a lot to know how each item is defined. For instance, how do they deal with AOL's dynamic IP addresses when computing a single access page? How are the individual browsers defined? It seems like the main report engine uses a different method than the filters section.
And the frustration comes because the Help menu describes why you would want to USE the numbers, but not what the number actually count.
ANOMALIES
My biggest reason for wanting documentation is some of the strange anomalies I see in the reports. For instance, I see a page with just 1 hit all week in the general report. Run a filter to EXCLUDE a particular browser and now I see over a thousand hits. This does not compute, and so I turn to the documentation, assuming I don't understand something or other and I get a pep talk about why I should care about this information rather than how the info is extracted from the raw log.
I also notice that I need to take great care interpreting the Stickiness Reports that come from a filtered run. Thinking about it, this only makes sense. If I filter out a pw protected directory, then user sessions that go directly from the Home Page to that directory will look like one-hit wonders. But I would have appreciated a little guidance on this.
Nevertheless, the greatest value of log analysis comes from seeing the big picture and from trend spotting. It's not about zeroing in on the oddities. But I would have more confidence in all the numbers if I could figure out where those weird ones come from. Ah, maybe with a little more time.
AND/OR FILTER COMBINATIONS
I wrote about this issue a few weeks back, commenting on version 2. Combining include and exclude filters with "and/or" rules is a bear. There's only a global switch - it affects every report definition for every site. And, you can't see it from inside the individual report to remember how it's set. In that thread, a rep of Fast Stats said they were considering changing it with this upgrade. But not so far.
SPIDER & BROWSER FILTERS
The biggest frustration is the fact that only some search engines and some browsers are globally defined. On any individual report you create, you can add custom strings to define more, but they need to be entered or copied all the time instead of being added to the global lists.
CAN'T ADD NEW SEARCH ENGINES
This one they've just GOT to deal with soon -- please? It's nice to see Teoma in the new standard SE list, but I get a lot of traffic from second tier engines, and I'd love to add them so their numbers end up in the standard Search Engine Performance reports.
SUMMARY
There are lots of little usability tweaks I'd like to see as well, but rather than continue with my laundry list and give you the impression I don't like the package, I think I'll stop. Because, truth be told, I really love it. I liked version 2 and it was a lot less powerful than this new baby.
There is a free trial, and it is worth checking out if you're looking for an analysis package. FastStats just may do everything you need. I can tell you that I've had version 3 for just one week, and it's already clarified a major business issue for a client.
-Ted
One user told me he went through his log, manually counting all referrers from search engines (only the ones defined in FastStats) and did not end up with the same number as reported - but higher.
As I said, I have not had the time to test this or varify it but it's worth looking into ...
My conclusion has always been as I said above - that the most important use of log analysis is watching trends, rather than getting absolutely solid numbers. Translating business language questions into software code is quite a sophisticated art.
For instance, search engines make constant shifts in their online offering. This makes "on the nose" accuracy very challenging for any log analysis package.
Just one example: if you get a hit from MSN preview (the nifty screenshot of your page) the referer info no longer shows the search term. How is a stats package going to handle that situation? Obviously it's a judgement call - one of many, many such calls required to produce reports.
This kind of situation is why I beg Mach5 to consider more complete documentation of how each item is defined. I'll continue to use Grep and Excel to explore anomalies that seem far off, and I will post about anything I find that would mis-direct business decisions. But, to date, the answers that FastStats gives are so valuable that they offest the anomalies by a wide margin.
My guess right now is that there is a learning curve on how to get more sophisticated answers - and that each website throws up its own peculiar obstacles to any stats package.
That's why (one more time) I really want to see better documentation.
If FastStats just count 10% lower all the time then the problem is not that bad, but what if the pattern is different ...? What if it only count one referrer for each hour - or cluster them in some other way that will NOT provide the consistency we need to track trends.
I will have to look deeper into this before I can say how big the problem is
If your count is different, try running the report again.
As for version 3? Well...I haven't upgraded. I don't think I will in the NEAR future. Site stickiness is not something that matters a whole lot to me.
"Average page views per day" is something I'd like to see, and perhaps is included with the upgrade.
But in all reality, I'm not ready to jump on this upgrade...I'm just too new to this 'log analysis' stuff and the extra stuff is just that...stuff.
I'm sure it's a great update, I just don't see the value, currently. (But that's just me.)
We offer each client the option of tracking a set of specific business questions, so I often run 5 to 10 differently filtered reports back to back. Somewhere in the area of report #8 to #10 I may need to close FastStats and reopen.
Not very comforting :)
A wise man said: You can't manage what you can't measure" but if the business decissions is made on wrong data it would be better to have no data and just use your instincts
That's how I turned up the MSN "preview" discrepancies. Because there is no record of the search phrase in the referer, those MSN hits don't get counted as a search engine hit. But they do show up in the Referring Domains report.
I can't remember ever getting two different numbers for the same item on the same report and logs. If that ever happened, I'd exit and reboot. Then run it twice and compare the two results. I'd bet it stays the same - but who knows.
I'll keep an eye out for that.
Nick
