logrotate rotates and compresses but doesn't continue logging

access.log stays at 0 bytes

[jamie]

hi,

we have been having logging problems with our redhat 7.2 server.

logrotate correctly rotates and compresses the logs at 4.02 am but then the access.log stays at 0 bytes with a last modified time of 4.02 am.

if i reboot the server then the logging starts again. (see previous post about apache crashing for my not being able to use the /etc/rc.d....http restart command).

there is an entry in /etc/cron.daily to run logrotate at 4.02 am

here are my files:

/etc/logrotate.conf

weekly
rotate 1
create
size=1M
compress
include /etc/logrotate.d
/var/log/wtmp {
monthly
create 0664 root utmp
rotate 1
}

/etc/logrotate.d/apache:

/var/log/httpd/access_log /var/log/httpd/agent_log /var/log/httpd/error_log /var/log/httpd/re
ferer_log {
missingok
sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true
endscript
}

I have also added a log script for every virtual server to the folder /etc/logrotate.d/
------

I am sure it has something to do with apache, as the access.log is reopened after the rotation, but is not being updated?

Any had similar experiences? Or could it be part of the problem with httpd not restarting well?

Many thanks!

[jpjones]

In my experience I've found that if you change the log file in any way (e.h. move it or compress it), then Apache will not add to the old location without a restart.

Probably something due to the way with which Apache writes to the file - e.g. it keeps the file open and simply writes in a stream rather than opening, writing, and closing the file for every request, which would be rather slow.

In short:
Everytime a log rotate is performed, apache has to be restarted.

HTH,
JP

[Gorufu]

logrotate correctly rotates and compresses the logs at 4.02 am but then the access.log stays at 0 bytes with a last modified time of 4.02 am.

if i reboot the server then the logging starts again. (see previous post about apache crashing for my not being able to use the /etc/rc.d....http restart command).

Apache needs to be restarted before logging starts again, so it is possible that the two problems are related.

/etc/logrotate.d/apache:

/var/log/httpd/access_log /var/log/httpd/agent_log
/var/log/httpd/error_log /var/log/httpd/referer_log

Wildcards * can be used in logrotate scripts.
/var/log/httpd/*_log will rotate all the above files.

The same applies for virtual hosts if they are all set up with the same directory structure.

For example /home/*/logs/*.log should rotate all the following

/home/host-1/logs/access.log
/home/host-1/logs/error.log
/home/host-2/logs/access.log
/home/host-2/logs/error.log

/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true

Run the above command through Telnet/SSH and see if apache restarts. The following command can be used to check that apache restarted.

tail -20 /var/log/httpd/error_log

You should see the last 20 lines and a notice saying apache restarted
configured -- resuming normal operations

I have also added a log script for every virtual server to the folder /etc/logrotate.d/

This should not be necessary if your virtual hosts have the same directory structure. Use wildcards and keep them in the same file /etc/logrotate.d/apache

Also check for syntax errors. A missing curly bracket could be causing the problem.

============================================

This is my /etc/logrotate.d/apache script

/home/*/logs/*.log {
weekly
missingok
rotate 4
}

/usr/local/apache/logs/suexec.log {
weekly
missingok
rotate 4
}

/usr/local/apache/logs/*_log {
weekly
missingok
sharedscripts
rotate 4
postrotate
/bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true
endscript
}

[jamie]

hi gorufu,

many thanks for the clear explanation. i think i have found one of the problems. i run the script to restart apache and voila:

/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true
cat: /var/run/httpd.pid: No such file or directory
cat: 2: No such file or directory

ahaaaa - so apache is not being restarted and so none of the freshly rotated logs can be updated!

well i locate httpd.pid and try again with the new command

/bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true
cat: 2: No such file or directory

hmmmm,
even when just doing
kill -HUP `cat /usr/local/apache/logs/httpd.pid`

it returns ok but i see nothing in the error_log to indicate that anything has happened.

obviously i just need to find out the correct way to restart apache and i think the problem should be solved?

still confused, but getting closer to the root ;-)

thanks

[Gorufu]

many thanks for the clear explanation. i think i have found one of the problems. i run the script to restart apache and voila:

/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true
cat: /var/run/httpd.pid: No such file or directory
cat: 2: No such file or directory

ahaaaa - so apache is not being restarted and so none of the freshly rotated logs can be updated!

well i locate httpd.pid and try again with the new command

You should be able to restart apache from /etc/logrotate.d/apache if you change
/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true to
/etc/rc.d/init.d/httpd restart

Hopefully that should fix the problem. You can easily test it before the normal log rotation time with the command

logrotate -f /etc/logrotate.d/apache

well i locate httpd.pid and try again with the new command

/bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true
cat: 2: No such file or directory

Your httpd.pid is probably located in /var/run directory for an Apache RedHat RPM installation so the above path would be incorrect on your server.

I have two custom compiled Apache servers running and two httpd.pid files, so they needed to be in different locations.

Have you sorted out the other problem?
restarting apache using /etc/rc.d/init.d/httpd restart

[jamie]

hi gorufu,

Have you sorted out the other problem?
restarting apache using /etc/rc.d/init.d/httpd restart

the httpd exec is locate here /usr/local/apache/bin/httpd

and i can restart apache with no problems by using this path. we're getting there ;-)

so now i only have to swap /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true

for /usr/local/apache/bin/httpd restart

a question:

won't that then throw out anyone logged into my site? isn't the -HUP command needed to ensure that each process is given time to end and that apache simply rereads its config files?

we have a fairly international visitor spectrum, so there is no one good time of day to 'cold' restart apache. or am i mistaken in my understanding of 'restart'?

many thanks

[amoore]

It's my understanding that restarting apache through your init script makes it stop serving pages for a time while it stops and then starts back up. Sending it a HUP signal makes it re-read its configuration file and stuff like that, including closing and reopening (log) file handles. It doesn't stop handling requests that way.

Therefore, it may be in your best intersted to learn how to HUP apache instead of restarting it.

[jamie]

i am getting somewhere.

i have just edited the httpd.pid location in apachectl to

/usr/local/apache/logs/httpd.pid

and for the first time, "apachectl graceful" works!

gorufu:

could i then use the /usr/sbin/apachectl graceful command in my /etc/logrotate.d/apache? doesn't this have a similar effect to -HUP

thanks all

[Gorufu]

HUP also kills processes. I have been doing some testing with a 100k animated gif and found that the request is killed using HUP or a full apache restart

The following commands using -HUP stopped the animated gif from loading completely.

/bin/kill -HUP httpd
/bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true

However, I found that a graceful restart does not kill any processes and the log files appear to restart without problems.

the httpd exec is locate here /usr/local/apache/bin/httpd

and i can restart apache with no problems by using this path. we're getting there ;-)

so now i only have to swap /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null ¦¦ true

for /usr/local/apache/bin/httpd restart

Not quite there yet. /usr/local/apache/bin/httpd is the apache httpd daemon and I would not recommend restarting apache that way, especially if you have any SSL vhosts.

You appear to have a compiled, non RedHat, installation of apache judging by the location of your apache files.

Have a look in the /usr/local/apache/bin directory and see if you have a file apachectl. It is best to start/stop/restart apache using that file if you have it.

/usr/local/apache/bin/apachectl will list all the available options.

To do a graceful restart try the following command.
/usr/local/apache/bin/apachectl graceful

If it works, then you can use it in your apache logrotate script.

<added by Gorufu>
Jamie, looks like you were on the same wavelength and figured it out while I was doing a lot of testing. Yes it should work without problems.

[edited by: Gorufu at 11:14 am (utc) on Feb. 21, 2003]

[Gorufu]

could i then use the /usr/sbin/apachectl graceful command in my /etc/logrotate.d/apache? doesn't this have a similar effect to -HUP

You seem to have an unusual installation of apache if apachectl is located in /usr/sbin directory and not /usr/local/apache/bin

Use /usr/sbin/apachectl graceful if it works.

Also check your sticky mail in a few minutes.

[amoore]

"apachectl graceful" sends the apache process the USR1 signal, so you can probably replace HUP with USR1 in your kill command if you prefer doing it that way.

[jamie]

hi amoore,

i think i've got it sussed now, using the apachectl graceful command. just ran logrotate -f /etc/logrotate.conf and everything rotated, compressed and started logging again :-)

cheers