logfiles based on query string

[VectorJ]

Is there any way to configure apache to write to a logfile based on the query string in a URL? For example, if a hit comes in with the URL [example.com?origin=foo&r=29&ip=37,...] can I have apache write the log entry to the foo.log file?

[sitz]

Yeah, you can, but it's a little ugly. Taking queues from the CustomLog documentation at [httpd.apache.org ] and a previous post in this forum [webmasterworld.com]:

RewriteEngine on
RewriteCond %{QUERY_STRING} origin=foo
RewriteRule ^.* - [E=foo-request:1]
CustomLog /tmp/foo.log common env=foo-request

...and if you want to ensure that 'origin=foo' requests do NOT get logged in the 'regular' access log, append an 'env=' snippet to the CustomLog directive used by your regular access log like this:

CustomLog /path/to/access.log common env=!foo-request

If you wanted to capture more than one query string, you could extend your Rule to:

RewriteEngine on
RewriteCond %{QUERY_STRING} origin=(foo¦bar¦baz)
RewriteRule ^.* - [E=%1-request:1]
CustomLog /path/to/foo.log common env=foo-request
CustomLog /path/to/bar.log common env=bar-request
CustomLog /path/to/baz.log common env=baz-request

(as always, replace the ¦ with a 'pipe' character)

Of course, another question is "do you *really* need to do this?" I know it's convienent having multiple logfiles, but this same functionality could likely be accomplished using post-processing on the logs (rotate the logs once a day, run a parser on the logs and split them up any way you want). I ran a fairly large virtualhosting system for a few years (several thousand VirtualHosts), and post-processing the daily logs so that each vhost had its own logfile was really the only way to go. Not saying that multiple logfiles in cases lke yours are *never* the right way to go; just asking if you're sure it's the right way in this case. =)

[VectorJ]

Thanks for the thorough response, it is greatly appreciated.

There is a better than average chance that what I need to do could be accomplished in a more efficient way. The reason for having the seperate logfiles is to provide real time stats to my customers; it'd take a long time to process the main logfile every time someone checked their stats. Due to the stats being real time, parsing and rotating the log once a day wouldn't work.

The other two options I've come up with are to give each user a subdomain or to pipe the access log to a handler that would update individual logs as appropriate. Any opinion on those approaches? Can you think of an option I've overlooked?

[sitz]

Individual subdomains are an option, but this will (also) result in several (or 'many'. Or 'thousands', depending on the nature of the site). A few tens (or even a hundred) sites with individual logs will work, but as the number of open logs increases, the number of file descriptors Apache has open will also increase; and yes, there is an ultimate limit. A handler is is another way to go, but potentially (depending on the nature of the handler) suffers a similar problem; a process can only have so many files open before Bad Things(tm) happen.

Have you considered a solution like mod_log_mysql [bitbrook.de] or mod_log_sql [grubbybaby.com]? The one you choose to use will depend a bit on the version of Apache you're running, but this solves several of your problems at once (of course, it creates new ones in the process. =) ).

You'd have to write (or find) a script of some sort your users could use to query the database, it would need to be secure enough that user1 wouldn't be able to access user2's stats, and you'd need some method to flush older data out of the database. One potential solution to this issue would be to have a cronjob that fires once a day, pulling data out of the database and placing it in a round-robin database [people.ee.ethz.ch].

Does all this help, or just increase the chaos? =)

[VectorJ]

For what it's worth, my solution to this problem was to use a perl module, File::Tail, to read a common file and split it up on the fly into individual log files. Mad thanks to sitz for helpling me get my thinker around this one.