Linux server running out of space

[csdude55]

I'm running CentOS 6.10 xen hvm with WHM / cPanel. It has a 100G SSD, but I'm constantly running out of space.

It's partitioned like:

Filesystem   Size Used Avail Use% Mounted on
/dev/xvda2    99G  92G 2.1G 98% /
tmpfs      1.9G   0 1.9G  0% /dev/shm
/dev/xvda1   248M  79M 157M 34% /boot
/usr/tmpDSK   3.2G 206M 2.8G  7% /tmp

I'm constantly removing the .GZ files from /var/log/apache2/archive, and there doesn't seem to be an automatic way to just not have them.

I also have a few .GZ files at /var/log, but they're only about 10M combined so not as major.

Any suggestions on what else I can remove to clear up some space?

I'm also pretty confused by the /domlogs/ section. When I go to /var/log/apache2/domlogs I have files like "domain-ssl_log". But then I go to /var/log/apache2/domlogs/ACCOUNT and have the same, identical file! In my biggest site's case that one file is 300M, and I obviously don't need it twice. But I can't figure out how to change it to only have one copy.

[graeme_p]

domlogs seems to be a cPanel thing.

In general, set up a cron job to delete old files. Be careful, and test what you are deleting. Something like this should work:

find /var/log/apache2/archive/* -mtime +7 -type f -delete

and delete files more than 7 days old.

I would test without the -delete, to see what would be deleted, then create a cron job for it.

[robzilla]

Any suggestions on what else I can remove to clear up some space?

yum install ncdu
ncdu /

:-)

[csdude55]

I tried that a few days ago, @robzilla, but unfortunately I don't know what's what :-( And it took HOURS to run.

Obviously the biggest hog is the /home directory, but I don't really want to mess with that. But there are so many log files and log backups that are spread out to a bajillion directories that I just don't know what can be safely deleted or cronned to auto-delete... some with a .gz or .old extension, some that are just impossible to determine from an automated perspective... like "messages-20201024". Just looking at /var/log I have 547M of files like that, not even looking in subdirectories :-O

And I don't even know what half of the files represent... anaconda, btmp, dmesg, dracut, tallylog, wtmp, never heard of them and will likely never look at these logs.

/var/log/audit/... no clue what it's auditing, but the files take up 25M. And the most recent one is 9/29/20.

/var/log/bandwidth/... seems irrelevant, the files are from 2016. Is this something I can delete, or does /var/log/bandwidth/ipmap somehow represent a setup that I don't remember making?

/var/log/dcpumon/... there are a bunch of toplog.[0-9]+ files that were modified today so I guess it's relevant, but it has subdirectories of /var/log/dcpumon/2017, /2018, and /2019, each with subdirectories for each month. Can I delete those? What about January - September in the /2020 subdirectory?

/var/log/sa/... what is that? The files are all like "sa01", "sa02", etc, and take up 17M. But they appear to be compiled and not plain text, so I have no clue what they're for.

What about the files under /var/log/munin/? I do use Munin to troubleshoot speed issues (maybe a few times a year) so I want to keep it going, but are the files here relevant or just taking up space? That directory alone is almost 4G.

See what I mean, there are files all over the place, some are important and some aren't. But what's what?

I wish there was a way to just stop backing up log files entirely. In 17 years, I've never once needed to look at a log file that's more than a day old... and rarely that's more than a few hours old. I DEFINITELY don't need exim_rejectlog-20201004 from October 4, or yum.log-20180101 from November 2017, or dracut.log-20150101 from March 2014 :-/

[robzilla]

It finishes in perhaps 10 seconds for me on a 120GB SSD. But that server isn't busy.

If you're not interested in the /home/ directory, excluding it will speed things up a bit:

ncdu --exclude /home /

Log file names like yum.log-20180101 suggest there's log rotation, but no compression. Most applications should do that by default, but check the rotation files in /etc/logrotate.d/ to be sure. And for the biggest log offenders, set the rotate parameter to a number of days you feel comfortable with.

For applications that don't have specific settings in /logrotate.d/, check the /etc/logrotate.conf file for default settings. The "compress" setting may be commented out by default.

As for Munin's logs, they're just that, logs. Helpful for troubleshooting but not very interesting otherwise. At 4GB, that would be an easy win.

What does du -hs /* look like for you?

[phranque]

When I go to /var/log/apache2/domlogs I have files like "domain-ssl_log". But then I go to /var/log/apache2/domlogs/ACCOUNT and have the same, identical file!

it may be a symbolic link to a file rather than a duplicate of the file.

if you do a 'ls -l' command the file mode of a symbolic link will start with an 'l' instead of a '-'.

[csdude55]

Log file names like yum.log-20180101 suggest there's log rotation, but no compression. Most applications should do that by default, but check the rotation files in /etc/logrotate.d/ to be sure. And for the biggest log offenders, set the rotate parameter to a number of days you feel comfortable with.

For applications that don't have specific settings in /logrotate.d/, check the /etc/logrotate.conf file for default settings. The "compress" setting may be commented out by default.

As for Munin's logs, they're just that, logs. Helpful for troubleshooting but not very interesting otherwise. At 4GB, that would be an easy win.

Awesome, thanks for the info! Nothing for Munin was in logrotate.d, so I did some research and modified logrotate.conf to include:

/var/log/munin/munin-graph.log
/var/log/munin/munin-html.log
/var/log/munin/munin-limits.log
/var/log/munin/munin-node.log
/var/log/munin/munin-update.log {
  daily
  missingok
  rotate 7
  compress
  notifempty
  create 640 munin munin
}

I also changed it to rotate daily instead of weekly, to keep 1 week of backlogs, and to compress by default. Hopefully that will help some :-)

What does du -hs /* look like for you?

Funny, this ran a LOT faster than ncdu! For some reason, though, it threw 12 errors like this:

du: cannot access `/proc/22760/task/22760/fd/4': No such file or directory

But otherwise, my biggest issues are /home (of course), /usr, and /var:

# du -hs /*
0    /aquota.group
0    /aquota.user
24K   /backup
8.2M  /bin
69M   /boot
128K  /dev
4.0K  /error_log
39M   /etc
91G   /home
405M  /lib
27M   /lib64
8.7M  /lost+found
4.0K  /media
4.0K  /mnt
258M  /opt
0    /proc
0    /quota.group
16K   /quota.user
4.0K  /razor-agent.log
673M  /root
16M   /sbin
0    /scripts
4.0K  /selinux
4.0K  /srv
0    /sys
27M   /tmp
8.9G  /usr
22G   /var

it may be a symbolic link to a file rather than a duplicate of the file.

You would think! LOL But alas, it's an honest-to-God copy, not a symlink. And I have yet to figure out why I need 2 copies, or what the purpose of either of them are.

[brotherhood of LAN]

>du: cannot access `/proc/22760/task/22760/fd/4': No such file or directory

Those are file descriptors used by running processes that were in the dir when it was read but no longer there when it came to measuring its file size. Or along those lines. Worth excluding that dir when using du etc.

Maybe worth looking at /home a bit more and seeing if anything non-important can be compressed or moved off the server.

I usually use Debian/apt but something like yum clean all might free some space up.

[JorgeV]

Hello,

You decided to self manage you server, so you are supposed to know everything about it, and which process does what, and where.

Now, as you mentioned it, it's your /home which is taking all the room. I would still inspect it to find out why.

For example, if you are using third party CMS, it's possible they create log files, backups, etc...

Also, it's possible that, if you allow visitors to upload images, there might be a folder holding all these images, or, "temporary" files, no longer used.

The importance of inspecting /home is also because, your server may be compromised somehow. Hackers might be or have uploaded files into your server, without you knowing it.

You need to know what your cron is doing too. May be a script or more are producing files, which are filling your SSD.

[robzilla]

I'd slim down /usr/ and /var/ first for some immediate relief, and then (as mentioned above) take a hard look at the /home/ directory. You don't want to be fighting for disk space all the time. If it's mostly (user-uploaded) images taking up space there, look into compressing them. Use ncdu to drill down and find the worst offenders.

Alternatively, ease your mind by upgrading to a bigger disk. If that requires re-installing, CentOS 6 is fast approaching End-Of-Life (November 30, 2020) anyway.

[csdude55]

Maybe worth looking at /home a bit more and seeing if anything non-important can be compressed or moved off the server.

@brotherhood of LAN, my biggest culprit here are a few accounts that INSIST on keeping their emails on the server! One hosting account takes up 23G by herself >:-( I tried to get her to download them, or at least download the old ones, but she literally laughed in my face and said that if I insisted then she would just take her business elsewhere.

I'm in a position where pennies count, so I just had to let it go. If I'm forced to upgrade to a larger server because of her then it's not worth it, but for now I'm squeezing every penny I can find.

My personal site is the largest (37G, and 12G of that is MySQL), but it pays most of the bills so that's OK. And I know it's safe, most of that storage goes to images sent by Private Message.

The woman above is my 2nd largest (23G), then the 3rd largest is another client that keeps their emails on the server but it's only 4G. The 4th is 526M, and after that they get small enough that I can't really chase it down too much.

You decided to self manage you server, so you are supposed to know everything about it, and which process does what, and where.

Wellll, yes and no. It's with [a hosting provider] and is supposed to be semi-managed (eg, they do the heavy lifting while I take care of general software maintenance), but after they sold to IBM they're an absolute joke. I submit a ticket and I'm lucky to get a canned response a week later.

Also, it's possible that, if you allow visitors to upload images, there might be a folder holding all these images, or, "temporary" files, no longer used.

Unless my Wordpress users have such a directory... my main site does have a /cache/ directory like that, but I have a cron to delete anything older than 24 hours that runs daily. It does take up about 250M /day, though.

You need to know what your cron is doing too. May be a script or more are producing files, which are filling your SSD.

Any suggestion on where I might find such a thing? I know where MY crons are, but if there's something that CentOS or WHM runs then I wouldn't have any idea where to look.

Use ncdu to drill down and find the worst offenders.

@robzilla, I've been playing with ncdu a little tonight, it's running faster right now (3am).

I figured out that /var/lib/mysql is a the main reason that /var/ is a hog, due to the MySQL for my main site... not a lot I can do there without a major site overhaul.

/usr/ has 3.9G for cPanel... not much I can do there, either. There's also /usr/tmpDSK, which I understand is the literal /tmp/ partition... there's another 3.2G.

Under /usr/local/apache.ea3/ I have /domlogs and /logs. I use Easy Apache 4, so I think these are copies of the log files when I updated over a year ago? Any guess on whether I can just delete both of these files? Or maybe even the whole /apache.ea3/ directory? That'd save me over 1G.

Alternatively, ease your mind by upgrading to a bigger disk.

Yeah, that WAS the plan for this year... now I can't really afford what I have, so the upgrade isn't realistic anymore :-( At this point, I have to scrounge, dig, and squeeze, and just hope that things improve so that I can upgrade next year.

[edited by: phranque at 4:29 am (utc) on Oct 27, 2020]
[edit reason] forum Charter [webmasterworld.com] [/edit]

[brotherhood of LAN]

my biggest culprit here are a few accounts that INSIST on keeping their emails on the server! One hosting account takes up 23G by herself >:-( I tried to get her to download them, or at least download the old ones, but she literally laughed in my face and said that if I insisted then she would just take her business elsewhere.

Perhaps consider migrating her to another hosting package, or charge her more. Most shared hosts go by a "fair use" policy (because they typically say things are "unlimited") and 22GB on a small hosting package is fairly excessive. When there's a that amount of data lying around for a site/host then I think it's reasonable for the site owner to pay a bit more and go with a VPS or even dedicated, management of the service aside.

The threat of moving elsewhere will cost her more in the long run as she'll need to migrate all that data as well as find someone to take care of things for her, which takes time.

[robzilla]

Under /usr/local/apache.ea3/ I have /domlogs and /logs. I use Easy Apache 4, so I think these are copies of the log files when I updated over a year ago? Any guess on whether I can just delete both of these files? Or maybe even the whole /apache.ea3/ directory? That'd save me over 1G.

If you don't think you'll ever need to revert back to EA4, you can delete the folder. Maybe back it up offline just in case.

If you have the time (and interest), learning to self-manage a server could save you quite a bit of money. You can get a good VPS with 300+ GB of SSD space for < $100/mo, probably quite a bit cheaper than what you're paying [your hosting provider] now. I've always thought "semi-managed" services were a bit of a joke, it's usually not a lot more than installing updates every now and then.

One hosting account takes up 23G by herself >:-(

At the very least, I'd put a cap on her allowance. If you can't change her current usage, set a maximum of, say, 25G. That's assuming you can do so legally, of course.

[edited by: phranque at 4:30 am (utc) on Oct 27, 2020]
[edit reason] forum Charter [webmasterworld.com] [/edit]

[robzilla]

I meant revert back to EA3, of course, not 4. Besides, log files are only for storing output, they're not actually used anywhere. It's very unlikely that a program will break once you delete its log files. (Sometimes you do need to restart a service after deleting the log files for it to open new ones.)

[tangor]

You need to manage your "email leeches" ... I don't know of many (or any!) mail providers don't don't have a STORAGE LIMIT while allowing unlimited bandwidth/transactions. Nearly all also have a "delete older than 6 months" function as well. STORAGE and UNLIMITED have two different meanings. Enforce it!

Just my two cents.

[graeme_p]

my biggest culprit here are a few accounts that INSIST on keeping their emails on the server

Storage is cheap though. Would it be possible to attach hard drives to the server and store emails on those? Or move email to a separate server? I have a separate mail server running on a $5/month server (+ VAT so its close to £5/month) and I have a lot more storage than your worst client uses

I've always thought "semi-managed" services were a bit of a joke, it's usually not a lot more than installing updates every now and then.

So are quite a few managed services! I have come across a few who really do not know what they are doing outside a fairly standard setup.

[topr8]

i guess you already know that centos 6 is EOL on 30 novemebr 2020

[csdude55]

Perhaps consider migrating her to another hosting package, or charge her more... The threat of moving elsewhere will cost her more in the long run as she'll need to migrate all that data as well as find someone to take care of things for her, which takes time.

In my case, this is a client that I originally built her website. So is the client that's using 4G. The custom web design industry seems to be dying (if not already dead), but I still make the occasional dollar off of them by doing maintenance.

There's another company in town that builds Wordpress sites for $250 that includes a year of free hosting, and most of my old clients hired them when it was time to update. I checked, and they're reselling a [another hosting provider's] account that's 30G of storage for $2.75 /month! They're really just a couple of kids that do it as a hobby, and there's no way that I can compete with their pricing. So what'll happen is, she'll just hire them to rebuild her site and move everything, making me lose any maintenance income, too. She might not be happy afterward, but by then it's too late.

I'm tempted to just buy a hosting account with them and move her over.

You can get a good VPS with 300+ GB of SSD space for < $100/mo, probably quite a bit cheaper than what you're paying [your hosting provider] now.

Whoa, do you mind recommending such a place, @robzilla? I have a pretty good legacy rate at $98.75 /month for my virtual server now (which includes WHM/cPanel), but if I can triple the storage for the same price...

I really need to increase my RAM, too, though. I have 4G now and end up swapping for 14 hours /day.

I have a separate mail server running on a $5/month server

Double whoa, @graeme_p ! Do you mind recommending where one could get such a monster? I'm all for fixing this for $5 /month!

[edited by: phranque at 4:31 am (utc) on Oct 27, 2020]
[edit reason] forum Charter [webmasterworld.com] [/edit]

[csdude55]

i guess you already know that centos 6 is EOL on 30 novemebr 2020

@topr8, yeah, which is why I had planned to upgrade this year... then COVID. I hope it just means that there are no more updates, but everything continues like normal? No one has really been clear on that.

[NickMNS]

Whoa, do you mind recommending such a place, @robzilla?

I use one of the more popular VPS providers (I believe that there is WW rule against posting recommendations about these things), they offer
16 GB RAM, 6 Cores, 320 GB SSD, 8 TB transfer, 40 Gbps, 6000 Mbps for $80 / mo, as well as a $5/mo option 1Gb ram, 1 core, and 25Gb storage. Their main competitor offers almost identical pricing.

You also have an option to add block storage, $1/mo per 10Gb.

One neat feature they offer is one click server stack installations, so if you want a LAMP server, you select LAMP server and the VPS is provisioned with everything fully installed. All you need to do is enter passwords, for PHP, and MySQL. Easy-peazy.

I have gotten to the point where I use a 5$/mo VPS to develop on. If I want to try something new, I simply clone the VPS make the changes on the clone. If I like it, I keep going on that version and delete the original otherwise I revert back to the original.

[phranque]

Whoa, do you mind recommending such a place, @robzilla?

Double whoa, @graeme_p ! Do you mind recommending where one could get such a monster?

mod's note: please see the section on Hosting Questions in the Webmaster General forum Charter [webmasterworld.com]

[robzilla]

but if I can triple the storage for the same price...

This would be an unmanaged VPS, but you might not actually notice the difference between that and "semi-managed". As NickMNS said, they make it pretty easy these days with one-click installations (e.g. of cPanel) if you're not comfortable enough to set it up yourself. There are lots of VPS providers, but the two Nick refers to were also on my mind; one's based in New York, the other in Philadelphia ;-) Shop around a bit.

Personally, I always keep my own sites and those of customers on separate servers.

I hope it just means that there are no more updates, but everything continues like normal?

Yes, it will continue to work, of course. I'm sure there are servers running CentOS 4 or 5 still. But if bugs or security issues are found, you won't get a fix, so it becomes a bit of a ticking time bomb. I have also been postponing it, though... moving servers can be such a hassle.

[csdude55]

Mea culpa, @phranque :-( Might I suggest a policy adjustment to allow established users to discuss hosting companies? I understand the spam concern, but it's also a relevant topic of discussion in cases like this.

You guys prompted me to ask my server company, though, and they DO have an option to set up an additional 100G HDD for $4 /month (or a 25G for $1). I have no clue how that second drive would be web accessible, but that might be the way to go... it's a heck of a lot easier and cheaper than upgrading the whole server, anyway.

[phranque]

Might I suggest a policy adjustment to allow established users to discuss hosting companies? I understand the spam concern, but it's also a relevant topic of discussion in cases like this.

in the absence of a policy adjustment, i would suggest webhostingtalk.com which is a long-standing and well-established forum on that subject.

[robzilla]

I have no clue how that second drive would be web accessible, but that might be the way to go

Probably network-attached storage that you can mount. Could be a good short-term solution, although you should have ample space for a while if you've cleaned up the hard drive.

[graeme_p]

Do you really need cPanel? If your customers need to admin their domains, you need it (or something similar) but if you are the only server admin there are plenty of setup scripts, example config files and management tools around. If you need something like Apache with PHP and MySQL you might need only small changes to the default config anyway.

I find it easier not to use cPanel because once you need to configure something outside its GUI it makes like more difficult.

We cannot discuss providers here, but they are easy to find, read some forums that do allow them, do some Googling, and Google for comparison sites too.

I currently have:

1. A VPS with a mid size European provider for multiple small Django/Python websites (mid size means not well known, but big enough to have some big customers and servers in multiple locations). I have standard config fules I can copy and paste with minor adjustments for this, then run certbot for https.
2. A VPS with a big European provider running a Jitsi (video conferencing server) server - configured using a step by step guide from their documentation.
3. A VPS with the same big provider running an email server and one static site using Mail in a Box (a set up script plus management GUI)

I still have two sites (my personal blog, and one static site). The former will eventually be replaced with a static site generator, and both will move to VPS 1.

I have clients using three of the better known US based providers (all have locations around the world). I have a favourite, but all are fine and provide roughly the sort of pricing I was talking about.

[JorgeV]

Hello,

Since you are with "[a hosting provider]", you are necessarily paying a lot more than anywhere else. Their prices are terrible.

I have 4G now and end up swapping for 14 hours /day.

Swapping is not good, even with a SSD, without counting that this is exhausting SSD cells lifetime.

4GB for a dedicated server, this is really low today..

As said, you can easily find (good quality) providers bellow $100 with much higher specs. Or even around $50 with at least 16GB and more.

[graeme_p]

Typo above, it should say, "I still have two sites on shared hosting"

[graeme_p]

At $100/month you can get a fairly good dedi. I just looked at one for under £50/month (excluding VAT) with two 2TB hard drives and 32Gb memory - and you can replace that with two 400+ Gb NVMe drives and it brings it to around £55/month.

That is not from the cheapest provider I could find either (its just the one I looked at when wondering about running my VPSs on my own dedi).

[tangor]

Just curious ... have you run an optimize/compact on your database recently?

If very active it can sometimes yield great results...