Welcome to WebmasterWorld Guest from 188.8.131.52
The above query was run on a 46mb log file to extract the top 20 URIs. I won't post the individual results but below are the stats Log Parser gave me about the overall job:
Elements processed: 293109
Elements output: 20
Execution time: 11.15 seconds
I'm not sure what you mean about how it compares to an un-optimized db query. I mean, to do that I'd first have to parse the log to get it into SQL Server and then do my query, right? As for searching through the files using something that supports regular expressions, I'll never do it again. This is so much easier than that if you're familiar with SQL queries.
I'm thinking along the lines of inheriting the component into a .NET application; and utilizing the .NET regular expressions namespace. You could really do some interesting non-traditional inquiry and data-mining log operations with this combination. ;)
I think I'll give it a go this weekend... anyone have any ideas for some test cases they'd like to see it run/benchmark it on?
Here is the Log Parser documentation [microsoft.com] in Word document
Some other SQL examples:
Performance-wise it seems to be pretty decent so far... I parsed through two years of logs for data on the top 10,000 by request - sorted by return code (200, 404, 500, etc.) and # of requests; and kicked out an XML file of the results all in under 30 seconds. I know there will be an improvement in performance in a regular environment as well(I was running it in debug mode - watching it very carefully for anything out of the ordinary).
I had the thought that I might go the route of an ASP.NET application - have it read the logs/parse/display the results to a secure webpage; though this is a bit more of an undertaking than I'm currently looking for. We'll see though.