Server overload

[jeje]

Hi all,
I'm posting this in the apache forum although the topic is broader I think.
I've been asked to look at a site generating frequent 500 internal errors. The site is an apache/joomla/mysql install run on a dedicated linux server.
Looking at the server, I see extremely heavy load averages: around 30 in normal hours and up to 90 during busy hours.
I figure there is some sort of configuration problem but haven't been able to find where (apache, mysql, php). Note that mod_suphp is enabled and php scripts are run as site owner.
I'd appreciate your inputs.

Here are some infos:

Apache:

Apache/1.3.42 (Unix) mod_gzip/1.3.26.1a mod_log_bytes/1.2 mod_bwlimited/1.4 mod_auth_passthrough/1.8 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.8e-fips-rhel5 configured

Top output (2208 http connections at the time):

top - 10:30:55 up 28 days, 16:48, 1 user, load average: 44.47, 38.04, 36.38
Tasks: 285 total, 4 running, 275 sleeping, 0 stopped, 6 zombie
Cpu(s): 45.3%us, 9.9%sy, 0.0%ni, 43.6%id, 1.1%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 12462248k total, 5214072k used, 7248176k free, 409068k buffers
Swap: 2104504k total, 144k used, 2104360k free, 3732868k cached


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4597 thepppos 17 0 30568 13m 6840 R 95.9 0.1 0:00.49 php
4485 thepppos 16 0 100m 21m 7296 R 84.1 0.2 0:02.05 php
23401 mysql 15 0 2432m 366m 4296 S 68.5 3.0 614:01.70 mysqld
4567 thepppos 16 0 105m 24m 7560 R 54.8 0.2 0:01.90 php
4569 thepppos 16 0 108m 28m 7672 S 50.9 0.2 0:01.97 php
4589 thepppos 17 0 96252 13m 6932 R 47.0 0.1 0:00.33 php
4595 thepppos 17 0 0 0 0 Z 37.2 0.0 0:00.20 php <defunct>

mysqladmin proc stat

Uptime: 71024 Threads: 22 Questions: 17147915 Slow queries: 7466 Opens: 3723 Flush tables: 1 Open tables: 1950 Queries per second avg: 241.438

[robzilla]

It doesn't take much to overload a Joomla set-up. Are they doing any type of caching? By default, Joomla has caching disabled, I believe. You should be able to find the setting under Global Configuration > System. There are caching plug-ins available as well. What about PHP and MySQL caching? Have they enabled query and thread caching? Installed an opcode cache such as APC, xCache or eAccelerator?

Free memory is wasted memory, and you have 7Gb of it. Consider caching more aggressively (i.e. store more data in cache).

They need to reduce the number of HTTP requests. It took nearly a minute to load, if I'm on the right site. 30 JavaScript files and 43 stylesheets is ludicrous.

[lucy24]

Apache 1.3?!?! I'm tempted to say that's half the answer right there.

30 JavaScript files and 43 stylesheets is ludicrous.

Double ?!?! I'm not expressing doubt; I've seen sites like that too. Generally it's symptomatic of an all-around mess behind the scenes-- the www equivalent to slapping on another coat of paint instead of fixing the dents and scrapes.