Strategies for validating log file info.

How do you validate the info in your logs?

[CMwebmaster]

How do you validate the information in your log files? Mainly, just for a sanity & reality check...

I'm thinking of adding a script to a few pages that would increment everytime the page was executed. Or, maybe doing something that logs to a database. Has anyone done something like this? Any recommendations?

[bakedjake]

Mainly, just for a sanity & reality check...

Why do you believe that your logs are wrong?

[ScottM]

Log analyzers have been notoriously suspected as being somewhat sketchy at best. The nice thing about them is they are usually wrong the same amount all the time.

So if your log analyzer says 100 today, 120 tommorow, about the only thing you can rely on is your traffic went up 20%. In otherwords, it may REALLY have been: 107 today and 128 tomorrow.

It's a best guess, especially with larger files. And trust me...what you want is the BIG files:>)

[bakedjake]

OH! I was under the impression he thought his actual log file data (as in the hard text data on the disk) was wrong. :)

[CMwebmaster]

Yes, i should have clarified...I want to double check the analysis of the log files against another source...some sort of page counter or something.

And, yes, the log files are about 1-2gig for each day...so pretty good size...large enough to make it impractical to manaully scan, and large enough to doubt some of the data from the analysis.

So, back to my original question...does anyone have good ideas for validating some of this data? I was thinking of writing something to a database, but I don't want to force a big performance hit upon my visitors, either.

[larryn]

What I do when I'm suspicious, is to use a different log analysis tool. So if you normally use say Webtrends, try against Analog.

Each log analysis tool has its own idiosyncronicites, so you shouldn't expect matching numbers, but in the same range at least unless you pick tools that have radically different analysis schemes.