Huge web logs, which analyzer should I buy?

log analysis tool for very large web logs

[porschegruven]

Guys,

I'm tired of trying products that won't crunch my logs. I don't want to have to turn off images, throw away data over a month old, etc. I simply want an inexpensive solution to crunch my large logs. We generate over 1BB hits a month so needless to say we have big logs since they include refferer data, arguments, etc.

We get about 1.2BB hits a month. We're not super graphic intensive so there are alot of html pages there that we need to know more about from a traffic standpoint. We also get many uniques. I've tried a bunch of the shareware type log analyzers and they just can't crunch my logs. A perfect solution would be something that could crunch logs that were years old. For this I am certainly open to a MySQL backend. My server environment is all Linux so I need software that crunches .gz logs on a linux box using MySQL.

My budget is limited. I really want to spend less than $1,000 on the software but will listen to alternatives because I need to solve this problem.

I'm really tired of trying all the software that can't handle my web log files sizes. It's extremely frustrating to wait hours or overnight to see a small subset only to get output errors due to ram problems, log software issues, etc.

I really need some help here, can you guys make some suggestions?

[John Carpenter]

I'd just try AWStats and rotate the logs. If the log is too big, you need to divide it into smaller parts. That's inevitable.

[nkad05]

I wrote a log analyzer once to read the W3C format. It could handle log files of any size. Sometimes I don't think these programmers realize how robust a log file analyzer needs to be. There's no reason a program shouldn't be written to handle huge logs.

[cgrantski]

1.2BB hits means 1,200,000,000 page views a month or 1,200,000 page view a month? Or is a "hit" any kind of hit?

[porschegruven]

No a hit is a hit and a page view is a page view. Hits include all graphics etc which I want to also count. Page views are 40MM+ per month. The problem w/ all the solutions I'm testing is that they can't count my page views for a month even w/ graphic analysis turned off. I want something that gives real resulst. It looks like I just need to log everything to MySQL and then work the data from there. Sure i'll have a little data server farm for the MySQL but I'd rather do that then rely on software which doesn't work. Sane.com seems to have a good product but it requires big $$ at my level of hits.

[Angelis]

Webtracker (from sane.com) would manage it, you can download a trial version from them that works for 30 days, perhaps you could give it a shot before you buy it...

Its not so costly as long as you only get a 1 site license.

[porschegruven]

Have you tried it? On their site it says you can't get the version that will talk to MySQL for less than $7,500. Does the non MySQL version work well w/ large data sets?

[gregbo]

Analog can crunch large logs, even logs that are several years old. You may have to play around with the config a bit in case fields have changed.