Recommended Log File Size

How large a file before archiving

[netguy]

After wondering why it was taking so long to run WebTrends reports, I realized that I was maintaining log files on my server of more than 1GB on several domains.

Four of my sites are now accumulating about 200 meg per month each, so I now have to determine a monthly archive, or something (?) to limit the file sizes - yet maintain enough information to generate useful reports.

I have 76 GIG in hard drives, so I can leave a lot of information on the server, but even with a cable connection (about 2MBaud download), it takes 20-30 minutes to process each site.

Any thoughts on how to make this more manageable?

Thanks,

Steve

[bcc1234]

You can rotate them on monthly basis and get a copy to your desktop and then append to the master file.
That way you local copy will grow and allow you to generate all the reports you need. And at the same time, your downloads will be small, so your server's connection won't be under heavy load for long periods of time.

[netguy]

bcc1234... your recommendation of appending the monthly [archived] files and manipulating locally is exactly what I was looking for. I think I'll have to dust off my old UltraEdit disk, cause I think notepad may be a little overwhelmed. ;)

Thanks!

Steve

[bcc1234]

I would not use a text editor to append the files.
If you got a 3GB master file and you want to append 200MB monthly log to it - your desktop will stall for good if you try to use clip pad or open both of those files in the editor.
If you don't have a unix box anywhere close by, I suggest you install cygwin.

[netguy]

Thanks again bcc1234...

I just checked, and UltraEdit is suppose to handle 4GB+ files (by disk swapping) - and I have some pretty strong PCs with a couple gigs of memory - so hopefully there won't be a problem. I should be able to append most log files for a 12-month period, which shouldn't take more than a 2GB appended file per website.

If there is, it's now nice to know about Cygwin.
Thanks,

Steve