huge logs?

help a noob with oversized log files

[bbkid]

I am willing to admit that I know very little about running a website. My sites fairly small, its been up for about 6 months. In the past 3-4 months its been getting a decent amount of traffic.
My Apache\log\access.log file was over 200 mb at the end of the fourth month.
I stoped apache, renamed the log file, then started apache.

I have been doing this for the past couple of months, with log files > 100 mb at the end of each month.

I downloaded the first log file to my desktop (athlon-64, 1 gb ram) and it took notepad well over 5 minutes to open it.

I'd like to know a couple of things:
Is there any sofware (preferably free / open source) that can manage log files for viewing?
ie: break down the data into something more colorful and easy on the eyes.

How can the size of the log files automaticaly be limited / divided? (i assume this can be done automaticaly)

[sitz]

Generally, one schedules a process to run once every so often; anywhere from once a month for low-traffic sites to several times a day for extremely high-traffic sites. On *nix, this is accomplished by moving the file to access.log.old (or access.log.$DATE, or what have you) and then issuing a 'graceful' restart. With the exception of the 'multiple times per day' model (which most sites don't need), the logroll is generally performed whenever the traffic is lowest (or, optionally, at midnight).

Another alternative is to set your access log to a pipe, and pipe the incoming loglines to a script of some kind; see [httpd.apache.org ] for syntax and [cronolog.org ] for an example of such a script. I've not benchmarked this to see what kind of performance hit (if any) one takes by doing this.

As far as opening the logfile, going through it line-by-line isn't really the best way to examine your logfiles, unless you're looking for something very specific. Analysis tools exist; I'm rather partial to webalizer ([mrunix.net ]), although it does have it's limitations. It is, however, *fast*.

Does that help?

[bbkid]

sweet, thanks. webilizer can help me some.

the other part will be more useful when i learn linux.
i do intend to stop using windows at some time, but i dont know linux well enough yet to switch over.

but still, thanks

[Erwin_D]

Apache comes with a utility called RotateLogs. You have to edit httpd.conf to enable it. It pipes Apache's log output, so that it can start a new logfile each day (or any period you specify).

Here's my access log line, set to start a new logfile every day at midnight:

CustomLog "¦bin/rotatelogs.exe logs/access_%Y%m%d.log 86400 60" combined env=!dontlog

[bbkid]

sweet, im gona try that out
thanks

[Matt Probert]

200 mB? And you think you have problems <bg>

Our log files for one week are over 1 gB! Anyway, we use Analog for the analysis, and break the one large log file down into several sub-parts using an in-house perl program, listed here for you:

#!/usr/bin/perl

$n = 1;
$x = 1;
$source = "access_log";
$t = "access_log";

$target = $t . "." . $x;

open(TARGET,">$target");
open(FILE,$source);
while(<FILE>)
{
print TARGET "$_";
$n++;

if ($n == 500000)
{
close(TARGET);
$x++;
$n = 1;
$target = $t . "." . $x;
open(TARGET,">$target");
}
}

By amending the value $n == 500000 you can make each sub-part larger or smaller. The resulting files are named

access_log.1 access_log.2 &c.

Matt

[bbkid]

ok, Erwin_D, 86400 seems to be the number os seconds in a day, so multiplying that by 30 gives 2592000, so puting

CustomLog "¦bin/rotatelogs.exe logs/access_%Y%m%d.log 2592000 60" combined env=!dontlog

at the top of my httpd.conf should start a monthly log right?
well, I'm going to try it.
one other question, what is the 60?

this is a great community, thanks to everybody. I'm definately glad to be a part of this.

[bbkid]

actualy, never mind. apache wouldnt start with my modified line, so I tried copy pasting yours and it still wouldnt start untill I commented it out...
so i dont know...

the error was "Failed to restatr the Apache service", not that that helps any...

[Erwin_D]

bbkid,

Make sure to replace the original CustomLog line that is already there. If Apache won't start, see the error.log for the reason.

[bbkid]

ok. I commented out the other customlog line, uncommented mine, and it still wouldnt start.

error.log had this line:

[Thu Mar 03 15:34:37 2005] [error] (2)No such file or directory: could not open transfer log file c:/program files/apache group/apache/\xa6bin/rotatelogs.exe logs/access_%y%m%d.log 86400 60.

¦ = \xa6 (?)

[edited by: bbkid at 8:51 pm (utc) on Mar. 3, 2005]

[bbkid]

oops. you can edit stupid posts, but you cant delete them.

[cooldoug]

LOL, Matt. Same here, my logs are over 700 mb every 2 weeks. This is using up all my space, is there a way to make it save the file to another server, like a free one, that I dont pay for?