Typical Disk Usage on a server - is this normal.?

[alistairgd]

I have a VPS with 25Gb of space. It ran out of space recently and I could not understand this as the two sites on the server only took up 7Gb inc mail and sql.

Support tell me that the usage is as follows

7 Gb web for two sites
6 Gb backup
4.5 usr
4.2 var

They said the rest (about another 4Gb) is OS files and cPanel files - does it sound normal that the usr and var folders and almost 9Gb combined for two small sites?

[csdude55]

I feel your pain... been there, done that!

The source of your pain is likely going to be log files in /usr and /var. You'll want to make sure that nothing is blowing up due to an error somewhere, but if you've had the server for awhile then it could just have a ton of unnecessary logs laying around.

My notes on this:

When /usr/ gets full, delete unnecessary files from
/usr/local/apache/domlogs/
/usr/local/apache/logs/archive (just backups from /logs/)

As well as:
rm -rf /usr/local/apache/logs/error_log
rm -rf /usr/local/apache/logs/suexec_log
touch /usr/local/apache/logs/suexec_log
httpd restart (maybe /scripts/restartsrv_httpd)

(httpd restart = restart Apache)

Another log file causing a problem for me:
cd /tmp/
rm -fr Net.WhoisIP.log; touch Net.WhoisIP.log

When /var/ gets full:
Check size of eximstats database:
du -sh /var/lib/mysql/eximstats

To clear data:
mysql eximstats

delete from sends;
delete from smtp;

Delete log files:
cd /var/log
rm -fr exim*.1
rm -fr exim*.2
rm -fr exim*.3
rm -fr exim*.4
rm -fr exim_mainlog; touch exim_mainlog ; chown mailnull.mail exim_mainlog
rm -fr exim_paniclog; touch exim_paniclog ; chown mailnull.mail exim_paniclog
rm -fr exim_rejectlog; touch exim_rejectlog ; chown mailnull.mail exim_rejectlog

service exim restart

If you use WHM/cPanel then you can change your settings to make it rotate and discard files more quickly.

[alistairgd]

Well, it helps some to know that I am not alone. I have been back and forth with support all day trying to figure out why a 25Gb VSP plan runs out of space with a couple of small websites that don't even use 9Gb.

The crazy thing is, if I had left these sites on the much cheaper reseller account I had there would be no problem because on the reseller plan backups are not counted towards usage.

I only had the server for less than a year, so bloat is unlikely.

I am using WHM/Cpanel but it is still a learning curve for me in this environment. For example, yyou have helpfully directed me to look in usr and var folder however I cannot locate them. In WHM menu I see no filemanager/browser option, and the filemanager for the two cpanels (sites) doesn not seem to have those folder...?

[csdude55]

I'm not sure if you can access them via WHM, I do it in SSH. Have you downloaded Putty yet? If not, it'll change your life :-) It's similar to Windows DOS, you'll log in and use basic commands like "cd /usr" to change directories to "/usr", and then "ls" to list everything in there.

I have a note file with dozens of SSH commands I've picked up along the way for general maintenance.

[csdude55]

And yeah, I had the exact same experience. There's a learning curve with your own VPS or dedicated server, and you're trading a few things away for speed and stability. I recently upgraded to a VPS with a 100G SSD and a secondary 250G HDD. Luckily, HDD storage is relatively cheap so at some point it's smarter to add more storage rather than spend your time trying to save what you already have.

[robzilla]

Yes that's pretty normal.

The crazy thing is, if I had left these sites on the much cheaper reseller account I had there would be no problem because on the reseller plan backups are not counted towards usage.

On shared hosting, that also goes for the OS files, because the OS is shared among customers on the same server.

Access to the /var and /usr folders may require privileged access to the file system, which may not be possible from cPanel/WHM.

If you don't have experience with Linux (csdude55's suggestions require command line access via SSH), I'd suggest you leave the system as-is and simply upgrade to a plan with more disk space. Or go back to the reseller plan if you don't require the benefits of a VPS.

[csdude55]

Rob, you made me realize... if he doesn't have access to /usr and /var, does he have any reason to care about log files, anyway? If not, a fast and immediate solution might be to ask the provider to delete those files.

Since I've been in his position, I'm guessing that he's desperately trying to clear up space because clients aren't getting emails...

[robzilla]

a fast and immediate solution might be to ask the provider to delete those files.

Fast and immediate, maybe, but not sustainable. They'll fill up again soon enough. Besides, I would expect log rotation is well-managed by cPanel and/or the host (assuming this is a managed VPS), so they're unlikely to be a problem.

It's normal for /var and /usr to be at least several gigabytes large (much of it OS-related). The /usr/bin folder, for example, contains most programs running on the server. MySQL, by default, stores its databases in /var/lib/mysql (may be different on a cPanel server, I don't know). I'm sure there's also quite a bit of fluff in /usr, but I'd only go culling that if you know exactly what you're doing, and even then it's probably not worth it.

Still, Google free up disk space on cpanel server and you'll probably find some ways to win back some mega-, maybe even gigabytes.

[not2easy]

A single terabyte HD can hold a lot of backups and being that they can be stand-alone, offline they are more useful if you ever need one.

[csdude55]

For now, @alistairgd, go to WHM > Tweak Settings > All.

Under "Stats and Logs":

1. I turned off "Archive logs in the user's home directory at the end of each stats run unless configured by the user"

2. I changed "Log rotation size threshold" to 10MB

3. I changed "Number of days to retain upcp logs before purging them" to 3

Under "Stats Program", I turned off all 4.

[csdude55]

@not2easy makes a good point about the backups. You can download those to your PC and then delete them to buy yourself some time.

Long term, a backup stored on the same drive as the original isn't ideal; a faulty hard drive means you lose the original AND backup! You might want to set up a second 10G drive with your VPS for those backups, anyway. You don't need WHM or cPanel for the backup drive, just a small bit of storage.

Or if you don't NEED to have regular backups (eg, you're dealing with static sites that rarely change) then you might could download and delete the backups, then turn off the backup option in WHM altogether.

[alistairgd]

Hi CSDude - thanks for the help so far, much appreciated.

> Have you downloaded Putty yet? If not, it'll change your life :-)

I have used it in the past, a little. In WHM though there is a "Terminal" option - I think this launches me into the same environment to make those commands?

> I recently upgraded to a VPS with a 100G SSD and a secondary 250G HDD. Luckily, HDD storage is relatively cheap so at some point it's smarter to add more storage rather than spend your time trying to save what you already have.

Yeh, been there, trying to delete files to keep a site within it specs, horrible game that. I'm paying $700 USD / Yr for a 20Gb VSP with 15Gb offsite backup that the host takes care of. To purchase additional 15Gb backup space is going to be another $85/yr.

> Since I've been in his position, I'm guessing that he's desperately trying to clear up space because clients aren't getting emails...

Not quite, the site actually went offline because of out of diskspace due to a recurring backup and retention. I have changed the backup retention to reduce the space used but I'm still on shaky ground. I don't think I need to have an WHM backups running because I paid for 15Gb of backup storage as part of my plan. As its been explained to me by support, this storage is separate to my cpanel. I need to understand that better as it could mean I can switch of the WHM backups.

Thanks again.

[alistairgd]

HI Robzilla

> If you don't have experience with Linux (csdude55's suggestions require command line access via SSH), I'd suggest you leave the system as-is and simply upgrade to a plan with more disk space. Or go back to the reseller plan if you don't require the benefits of a VPS.

Upgrading beyond $700 is not an option - I don't feel it would be value either as this is one site I'm running on there and although it is a e-learning site I am wondering if it really needs a VPS. So you idea of taking this site back to the cheaper and easier to manage reseller plan really appeals to me.

> It's normal for /var and /usr to be at least several gigabytes large (much of it OS-related).

Okay, fair enough thank you for confirming.

[alistairgd]

> Or if you don't NEED to have regular backups (eg, you're dealing with static sites that rarely change) then you might could download and delete the backups, then turn off the backup option in WHM altogether.

Yes I like this idea, it's one site under 4 Gb, it's not a big deal for me to download manually, ie the web folder, the mail folder and the sql files. That way I can switch off the WHM backups.

[not2easy]

Via WHM you can use each site's CP to zip it into a little file, in your choice of bz, gz or tar format. it is a trivial size for storage. ;)

[robzilla]

Upgrading beyond $700 is not an option

$700 a year for a managed VPS with only 25 GB of disk space? You can probably find a better deal.

I assume it's a managed VPS. If not, you shouldn't be paying more than $200/year.

[alistairgd]

Hi Rob

> $700 a year for a managed VPS with only 25 GB of disk space? You can probably find a better deal.

Yes it is managed with immediate response online ticketing support. Support is important to me as some areas remain a dark art.

[alistairgd]

Hi not2easy

> Via WHM you can use each site's CP to zip it into a little file, in your choice of bz, gz or tar format. it is a trivial size for storage. ;)

In Cpanel for the site, the Backup section has a button to download a full site backup - I've never used it before so I will give this a go and can use this to make manual backups as a safeguard.

Thanks.

[graeme_p]

I agree with robzilla, it is expensive for a managed VPS, and very expensive if its unmanaged. You can get that much or more disk space with a $5 to $10 a month VPS.

If its managed they should take care of things like managing logs.

if its unmanaged you can get a dedicated server for that much.

In my opinion, its easier in the long run to learn the command line rather than rely on cPanel and WHM. There is more documentation, you have finer control over how things work, its easier to troubleshoot, and its instrinsically more secure less attack surface.

[alistairgd]

Thanks Graeme, the primary reason I moved this site of a reseller onto a VPS was for the increased security. I didn't really anticipate the management overhead and the time I'd have to spend on it. I agree with the command line advice, thanks.

[Mark_A]

I had this issue, I had a VPN with a number of sites on it. When I was in hospital, when I couldn't do anything about it, the server fell over - disk full - and some of my customer's email stopped working. It was log files.

[engine]

When I was in hospital

Hope you're all ok now.

the server fell over - disk full - and some of my customer's email stopped working. It was log files.

I have a rotating log file deletion to avoid those issues, which automatically deletes files older than (choose your own date).
The log files are automatically downloaded weekly to keep them locally.

[graeme_p]

engine beat me to it. Rotating log file deletion is essential.

Privacy laws also limit how long you can keep them for. For example, if you web server logs IPs you must should the IP address as personal information and delete it when you no longer have a legitimate reason to keep it.

[Mark_A]

I have a rotating log file deletion to avoid those issues, which automatically deletes files older than (choose your own date).
The log files are automatically downloaded weekly to keep them locally.

Yes, that was pretty much what I did when I realised what the issue was. And I zipped the logs also. If I had been around it wouldn't have taken long to resolve but because I was out of touch I lost a customer, nearly lost more. Both they and I realised the real risks in working with a one man band.

And engine I am ok now thank you for asking.