how can i reduce server load?

load average too high

[sunroof]

At peak times my server load averege is really high, it goes up to 30, and now it is cca 12, and server is really slow, web pages are barely opening. I host two sites with cca 30.000 unique visitors per day. Server configuration is INTEL-XEON-2x2.8 with 4GB of RAM, server os is CentOS_4.x, with Plesk.

Is there any way to reduce server load, or do I have to buy another one?

Here is the top output of my server:

19:03:35 up 2 days, 17:52, 1 user, load average: 12.74, 15.33, 11.65
229 processes: 225 sleeping, 3 running, 1 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 3.6% 0.0% 1.1% 24.7% 24.2% 40.8% 5.3%
cpu00 4.5% 0.0% 0.9% 0.9% 51.3% 21.1% 21.1%
cpu01 3.7% 0.0% 0.0% 0.0% 45.3% 50.9% 0.0%
cpu02 6.4% 0.0% 2.7% 0.0% 0.0% 90.7% 0.0%
cpu03 0.0% 0.0% 0.9% 98.1% 0.0% 0.9% 0.0%
Mem: 4093116k av, 3962768k used, 130348k free, 0k shrd, 114796k buff
2871260k actv, 793208k in_d, 65864k in_c
Swap: 1052248k av, 161568k used, 890680k free 1603076k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
30470 apache 15 0 0 0 0 Z 1.1 0.0 0:01 2 httpd <defunct>
17139 apache 25 0 2132 2132 1504 R 0.6 0.0 1:39 0 perl
25230 apache 15 0 12476 12M 6332 S 0.6 0.3 0:04 1 httpd
32046 apache 15 0 10748 10M 6308 S 0.4 0.2 0:00 2 httpd
2343 root 20 0 1252 1252 888 R 0.4 0.0 0:00 2 top
25 root 15 0 0 0 0 DW 0.2 0.0 45:49 2 kjournald
30454 apache 15 0 11628 11M 6332 S 0.2 0.2 0:01 0 httpd
658 apache 15 0 11020 10M 6284 S 0.2 0.2 0:00 2 httpd
2287 apache 16 0 8996 8996 5820 S 0.2 0.2 0:00 2 httpd
2345 mysql 21 0 1158M 1.1G 2668 S 0.2 28.8 0:00 2 mysqld
1 root 15 0 488 488 428 S 0.0 0.0 0:15 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
4 root RT 0 0 0 0 SW 0.0 0.0 0:00 2 migration/2
5 root RT 0 0 0 0 SW 0.0 0.0 0:00 3 migration/3
6 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
7 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
8 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
9 root 34 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd/2
10 root 34 19 0 0 0 SWN 0.0 0.0 0:00 3 ksoftirqd/3
13 root 15 0 0 0 0 SW 0.0 0.0 0:24 1 bdflush
11 root 15 0 0 0 0 SW 0.0 0.0 3:11 1 kswapd
12 root 15 0 0 0 0 SW 0.0 0.0 4:33 1 kscand
14 root 15 0 0 0 0 SW 0.0 0.0 0:43 0 kupdated
15 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
80 root 25 0 0 0 0 SW 0.0 0.0 0:00 2 khubd
1302 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
1303 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

Thanks

[zCat]

2345 mysql 21 0 1158M 1.1G 2668 S 0.2 28.8 0:00 2 mysqld

You very probably have some atrociously constructed mysql query which is eating globs and globs of memory and writing a humungous tempfile to disk, which will slow the whole system down. If so, the query needs to be optimised, because unless you're working with very large datasets your server looks like it should be running with a load of about 0.05.

[edited by: zCat at 5:19 pm (utc) on July 31, 2006]

[lammert]

CPU states: cpu user nice system irq softirq iowait idle
total 3.6% 0.0% 1.1% 24.7% 24.2% 40.8% 5.3%

This line makes me nervous. Only an average of 3.6% of your available CPU cycles are actually used for user processes (mysql, apache etc). OTHO, system, irq, softirq and iowait takes up more than 90% of the time.

This indicates a problem in your disk subsystem. Either your disk system is not capable of handling the load, or you have a RAID5 system where one disk went bad and the missing data must be recalculated every time with the remaining information on the other disks.

[arran]

The high iowait% suggests that either you are experiencing poor disk performance (use "iostat -x 4" to check) or that the system is paging due to lack of memory (probably caused by the mysql process pointed out above).

arran.

[sunroof]

@zCat: I'll check out my queries, although I have phpbb forum installed with 26532 registered users and 670.245 posts... average people online cca 80-100.

@lammert: I have IDE disk... If I change it to SCSI would it solve the problem?

@arran:
[root@ltct-1887 root]# iostat
-bash: iostat: command not found

Thanks a lot guys!

[sunroof]

here is 01.00 am where i live now, and load average is cca 1-2...

here is the output...

 00:54:57 up 2 days, 23:43, 1 user, load average: 1.07, 1.86, 3.51 
181 processes: 180 sleeping, 1 running, 0 zombie, 0 stopped 
CPU states: cpu user nice system irq softirq iowait idle 
 total 15.9% 0.0% 3.4% 3.4% 0.0% 7.3% 69.6% 
 cpu00 23.3% 0.0% 4.5% 0.1% 0.3% 3.1% 68.2% 
 cpu01 15.3% 0.0% 2.3% 0.0% 0.0% 3.1% 79.0% 
 cpu02 10.9% 0.0% 4.1% 13.7% 0.0% 5.3% 65.6% 
 cpu03 13.9% 0.0% 2.7% 0.0% 0.0% 17.5% 65.6% 
Mem: 4093116k av, 3961372k used, 131744k free, 0k shrd, 110440k buff 
  3031872k actv, 612636k in_d, 69872k in_c 
Swap: 1052248k av, 161568k used, 890680k free  1688932k cached 

 PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 
 2954 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 3:20 1 mysqld 
 2955 mysql 24 0 1156M 1.1G 2676 S 0.0 28.8 3:36 1 mysqld 
 2956 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 0:00 0 mysqld 
 2957 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 0:16 0 mysqld 
 2958 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 1:19 1 mysqld 
 2959 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 2:18 1 mysqld 
 2960 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 3:08 1 mysqld 
 2961 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 2:33 3 mysqld 
 2963 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 8:35 3 mysqld 
 2964 mysql 15 0 1156M 1.1G 2676 S 0.0 28.8 0:59 2 mysqld 
23075 mysql 16 0 1156M 1.1G 2676 S 0.0 28.8 0:18 2 mysqld 
25480 mysql 25 0 1156M 1.1G 2676 S 0.5 28.8 0:00 3 mysqld 
25522 mysql 25 0 1156M 1.1G 2676 S 0.3 28.8 0:00 3 mysqld 
25524 mysql 20 0 1156M 1.1G 2676 S 0.0 28.8 0:00 2 mysqld 
24583 apache 15 0 28308 27M 6580 S 1.8 0.6 0:00 1 httpd 
22965 apache 15 0 15856 15M 6664 S 0.0 0.3 0:02 1 httpd 
21404 apache 15 0 14680 14M 6680 S 0.0 0.3 0:03 3 httpd 
20736 apache 15 0 12556 12M 6736 S 0.0 0.3 0:01 0 httpd 
23386 apache 16 0 12288 12M 6664 S 0.1 0.3 0:01 0 httpd 
21084 apache 15 0 12184 11M 6728 S 0.0 0.2 0:01 3 httpd 
20828 apache 15 0 12168 11M 6668 S 0.0 0.2 0:01 2 httpd 
24559 apache 15 0 12156 11M 6604 S 0.0 0.2 0:00 1 httpd 
20905 apache 15 0 12152 11M 6708 S 0.0 0.2 0:01 3 httpd 
22539 apache 16 0 12104 11M 6696 S 0.3 0.2 0:01 3 httpd 
22538 apache 15 0 12092 11M 6704 S 0.0 0.2 0:01 1 httpd 
24325 apache 15 0 12092 11M 6660 S 0.0 0.2 0:00 0 httpd 
21385 apache 15 0 12072 11M 6668 S 0.0 0.2 0:02 1 httpd 
23420 apache 15 0 12072 11M 6644 S 0.3 0.2 0:00 2 httpd 
21408 apache 16 0 12044 11M 6884 S 0.1 0.2 0:01 1 httpd 

[jake66]

where are you able to get this information, is it available on a shared server? i was told by a host that my cpu / memory use is through the roof and requires a dedicated server, though everytime i look at my stats via phpinfo they over between 0.40 to 0.80 / 0.95

no specific queries were ever pointed out as offenders, so i do not know what to fix (if there is even a problem)

[sunroof]

I have dedicated server and shell access... I don't think you can get this information if you have shared server, although some hosting companies give you ssh access...