combining daily logs into a monthly file

combining daily logs

[mission]

I recently changed hosting companies.

I was accustomed to having one access log file per site (every day's info would be added to the same file).
My new host has a system that makes one log file per day (per site). This works with the analysis tools they provide (webalizer and modlogan).

This is a problem for me, because I want to analyze the files with a program that can only work on the one data file that I feed it at a given time (that is, if the log file has data for 3 months, it will analyze that; it it only has data for 1 day, that's all it will analyze).

My new hosting company is Worldzonepro, and its servers are run on Linux. The log directory keeps the daily log files for a week. Log files are identified with the site's name and a number (but the daily numbers are not in sequence).

Anybody has any suggestions for automatically combining the daily logs through Cron, or otherwise?

[dcrombie]

You should be able to use the "cat" command on multiple files and redirect the output into a new file:

cat logfile1 logfile2 ... logfilen > combinedlogfile

If the logs are compressed (.gz or similar extension) then you need to throw gunzip (or another decompression program) into the mix.

This is a common problem so there's probably scripts available if you start searching.

[mission]

Thanks.
Further questions based on that:
- I'd love suggestions for such a script
- Does this depend on having daily specific log file names (on my server, log files are identified with the site's name and a number, but the daily numbers are not in sequence, nor do they refer to the date).

[killroy]

I've written a program that takes apache logs and turns them into a fully indexed and normalized MySQL database, often shrinking it to around 10% of the original size as it avoids duplication.

To merge daily logs into a large log for processign into MySQL, I use a simply binary file merger. I've tried a few, and the one to work flawlessly was a program called Simple File Joiner.

This is on win32 though.

Alternatively you could log straight into MySQL using mod_logmysql (or something like that)

SN

[mission]

Thanks, killroy

I must admit this is well above my level of competence. I don't know:
- how to create a script that would instruct the server to bring in the log files to mysql
- how to use mysql
- or how to identify each day's log file (they have what seems to me like arbitrary numbers; except, of course, if I just asked the program to take the current day's file just a minute before the server closes it to compress it?)

Anyway, what I'm trying to say is I'm hoping to find somebody who is in the same case as I am, who could lead me through a detailed step-by-step approach of what they did, so that I could replicate it even without much technical know-how.

[feeder]

Tried AeroMerge?