Log files........ too big

unix log files compression

[JT_Clark]

I have a site that I host on a Unix dedicated server. It is the only site hosted on it. Gets between 150,000 - 250,000 page views per day, and obviously more hits to assorted things like graphics, cookies, style sheets, etc. All of which results in log files that grow daily to over 100 MB. I delete them and reboot at midnight each night, and the process begins again.

My problem is this is taking up almost all of my system memory. I need the stats so I can't just turn the logs off or remove them altogether. Is there any way to cut down on memory use without 1) removing or turning off logging or 2) adding more RAM?

Server specs:
Unix
1.13 Ghz processor
512 MB RAM

PHPSysAdmin consistently shows my memory use at 95-99% once traffic starts rolling in around 7 am. When logs are deleted and new ones created at midnight it drops down to 70% or below.

[bcc1234]

Configure your logs to be sent to a process instead of a file.
And the process that receives the data should be a pipe to gzip.
So you basically compress the logs as the come in. Just make sure you don't reboot http server wihtout renaming the file first. It won't be able to append to a compressed file.

Besides, what makes you think that log files take up ram?
Most unix systems use available ram for disk caching, so it's reported as being used, but in fact if any process needs more memory - the disk cache is flushed and the memory is reallocated.

Run vmstat during your pick hours are see the there is a lot of paging in going on. If not - you are fine.

[JT_Clark]

Not quite sure what it means, but this is what I get when I run vmstat:

  procs           memory  swap     io   system     cpu 
 r b w  swpd  free  buff cache si so  bi  bo  in  cs us sy id 
 0 0 0 10520  8652 104552 223292  0  0   0  11  14   5  7  2  8 

Also, PHPSysAdmin shows the following:

Memory Usage  
Type Percent Capacity Free Used Size  
Physical Memory  95% 22.68 MB 478.94 MB 501.62 MB  
Disk Swap  2% 507.41 MB 10.27 MB 517.68 MB  

[andreasfriedrich]

Welcome to WebmasterWorld [webmasterworld.com] JT_Clark.

Be sure to read Marcia`s WebmasterWorld Welcome and Guide to the Basics [webmasterworld.com] post.

man vmstat

will explain about the data that get displayed.

Andreas

[bcc1234]

You are fine.

[Duckula]

What bcc1234 means is explained on this [webmasterworld.com] thread.

[bcc1234]

He-he. We should post some links to basic how-tos and keep it as the top thread in the category.

[Josk]

Have you seen the site www.linuxdoc.com? Its avery, very, very good reseource for beginners and experts...

[dingman]

Big fan of sunsite/metalab/ibiblio, myself.

[bcc1234]

Have you seen the site www.linuxdoc.com? Its avery, very, very good reseource for beginners and experts...

I don't run linux for religious reasons, so that won't be of much help to me :)

[dingman]

I don't run linux for religious reasons

Are you one of those horrible people who think that only BSD-style licenses are really free? ;)

At least for beginner stuff, I think most things cary over. Ports is a bit different from RPM, to be sure, but then so is 'apt'. Both systems tend to have the same shells available and the same GNU utilities and mostly the same set of libraries and user apps will compile accross the freenixes.

[bcc1234]

I don't care about the license.
Although, I do belive that people who call GPL'ed software free are hypocrites, I have nothing against GPL.
If I were to release something to the public - it would be at least GPL'ed.

I just don't like the way linux-based systems are structured. It's basically a bunch of patches by a lot of people with different opinions about how things should be done.
And I believe that for any enitity there should be some authority that coordinates the mess and make final decisions.

[Duckula]

some authority that coordinates the mess

I can't speak for every distribution out there, but at least Debian has a very strict policy [debian.org] about where everything goes.

I can assure you, that is not a mess.

We are getting dangerously near to an offtopic flamefest here... let's stop. Or at least change of thread.

[dingman]

let's stop

Good plan. I do hope people realize that my use of "horrible people" was tongue-in-cheek. And that I was serious in suggesting that there might be a lot of documents at the new-user level that carry over pretty well between *nixes.

[bcc1234]

We are getting dangerously near to an offtopic flamefest here... let's stop. Or at least change of thread.

I doubt we are getting anywhere even close to the OS flame.
If I wanted that - I would go to #Linux on EFNet :)

I have many clients that run linux and I have to work with their boxes all the time. The problem is a lot of little things are done in different ways.

/etc/init.d/rc.d/
/etc/rc.d/

/etc/httpd/log
/var/log
/var/log/httpd

/usr/local/etc/apache
/etc/httpd/conf

/usr/local/bin/perl
/usr/bin/perl

/usr/local/libexec/apache
/usr/local/apache/libexec

Looks familiar?
And many-many-many other little annoying things.
And you sit in front of the terminal and say "Oh, man I thought that's how it's done here. Dammit, let me try the other file...."

The problem is not linux (it's a nice kernel), but the fact that the whole community is oriented towards people who can either be completely dependant on how-to's from a particular distro or be super-geeks that know every bit of every file on the box.

I'm neither of above. I need to get things done quickly, but still be flexible. That's why I don't use linux.

The reasonable thing for me to do would be to either use Red Hat or not use linux at all. So I'm just not using linux at all for my needs.

And that I was serious in suggesting that there might be a lot of documents at the new-user level that carry over pretty well between *nixes.

Oh yeah, on the user level - it's all pretty much the same thing.