/var/log 100% full but du reveals no big files...

any ideas on what is taking up all the space.

[tolachi]

Basically /var/log is showing as almost 100% full but I can't find any big log files in it to delete. It has 1 GB of disk space and I can't find anything in it over a couple of megs. Here is a du cut and paste:

chitown:/var/log# du -h *
32k XFree86.0.log
34M apache2/old
138M apache2
0 auth.log
0 daemon.log
0 debug
16k dmesg
4.0k dpkg-iasearch.log
0 faillog
0 kern.log
141k ksymoops
0 lpr.log
0 mail.err
0 mail.info
0 mail.log
0 mail.warn
0 messages
29k mysql
0 mysql.err
0 mysql.log
512 news
512 ntpstats
512 php4
949k qmail/smtpd
1.9M qmail
121k qmail-send
237k qmail-smtpd
0 setuid.changes
0 setuid.new.tmp
0 setuid.today
0 setuid.yesterday
929k svscan
0 syslog
0 user.log
0 uucp.log
8.0k wtmp
16k xdm.log

and a df cut and paste:

chitown:/var/log# df
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda6 2529920 52684 2348720 3% /
/dev/sda5 4000024 1028720 2971304 26% /usr
/dev/sda7 1999992 223472 1776520 12% /home
/dev/sda8 995960 32840 963120 4% /tmp
/dev/sda9 1999992 1044828 955164 53% /var
/dev/sda10 995960 32840 963120 4% /var/tmp
/dev/sda12 995960 32856 963104 4% /var/spool
/dev/sda2 93329 6050 82460 7% /boot
/dev/sda11 995960 995212 748 100% /var/log
/dev/sda13 1999992 175368 1824624 9% /var/log/apache2

Please let me know if there is any other info you could use. I have no clue how all of this space could magically be used up.

[mcavic]

The only thing that comes to mind is that if there's a deleted file that a process still has open, it'll still be taking up space. In that case, you could try rebooting. Seems a longshot, though.

[tolachi]

Interesting. Unfortunately rebooting on this box is a major pain. I think that will have to be my last resort. Any other ideas?

[moltar]

You can just restart the service. Try restart httpd first. Then look at mysql.

[py9jmas]

Try

du . ¦ sort -n

This will sort the directories in size order (note directories include the space taken up by subdirectories). The -h option is your enemy, it makes it harder to spot the one or two big items. Using du * will also skip any files/directories begining with a period.

[edited by: py9jmas at 5:09 pm (utc) on May 12, 2005]

[mcavic]

Yes, you can try restarting any service that might use /var/log.

[tolachi]

Fascinating. It seems that mysql must have been the culprit. Thanks for helping me out with this. I really appreciate everyone's advice.

[moltar]

Maybe your log level is set too high? Another thing also to look into is automatic log rotation.

[tolachi]

I don't think it was either of those because there were no even medium sized log files that could have been the problem. I'm pretty sure it has to do with some weirdness left over from trying to downgrade to debian stable about a week ago. And /var/log and contents are the right size now.