Apache (I think) performance problems

how do I find the bottleneck?

[muszek]

Hi. This is going to be a long post, so I'll try to write it in an organized manner.

The problem
My rather lightly used server sometimes spends way too much time serving pages. I can't really find the source of the problem, so I'll try to give you as much data, as possible. First, what happens. That's a sample output of wget:

wget http://domain.tld/a.html
--22:51:36-- http://domain.tld/a.html
   => `a.html.5'
Resolving domain.tld... xx.#*$!.#*$!.#*$!
Connecting to domain.tld¦xx.#*$!.#*$!.#*$!¦:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9 [text/html]
100%[====================================>] 9    --.--K/s    
22:51:36 (732.42 KB/s) - `a.html.5' saved [9/9]

When the problem occurs, wget "freezes" after "HTTP request sent, awaiting response..." (before "200 OK") for even as long as a minute.

circuimstances (hardware, software, etc.)
It's a AMD64 3200, 1GB server running on Debian Sarge with Apache 2.0, MySQL 4.1 and backported PHP 5.2. I installed the latest APC for PHP and I made a script that clears the cache once per hour. CPU is ALWAYS at least 80% idle, there's always some free RAM. Swap is almost never used (it wasn't the last time problem was occuring).
I have 3 main sites on that. One Drupal 4.7 site: ~20k impressions per day. One self-written, lightweight PHP: ~30k per day. One Drupal 5 site: ~3k impressions/day. And a dozen or so sites that receive almost no traffic.

It's a vanilla Debian - configured by myself. I use afraid.org (scared of configuring BIND... but as you can see above (wget output), the problem occurs after DNS-related work is done).

top (current state)

Tasks: 68 total, 2 running, 66 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7% us, 0.7% sy, 0.0% ni, 97.3% id, 0.3% wa, 0.0% hi, 0.0% si
Mem: 906736k total, 700996k used, 205740k free, 181760k buffers
Swap: 2650684k total,  4k used, 2650680k free, 264252k cached

What could I do to find the bottleneck? I'm not a professional admin, so I don't really know any tools. Just a regular Debian/Ubuntu user.

[muszek]

more details:
* restarting apache doesn't help.
* other services (ssh, pinging from outside) are fine.

[wheel]

Tough to diagnose these things. But a few things come to mind.

The first is to try increasing the number of threads or connections that apache allows.

Second, check your firewall, it could be doing something funky. I've seen hardware firewalls act stupid sporadically.

And the other thing I'd check next time the problem crops up is how many processes are running. From the command line do a 'ps aux' which will list all the running processes. If you've got a whack of apache or mysql threads that will point to the problem as well. (for example, if mysql gets backed up, sometimes the apache thread will get left hanging open as it waits. Then someone else visits and opens a new apache thread, which makes a mysql call which has to wait - leaving another apache thread open,and so on. )

[equalm]

I would echo Wheel's thoughts in that you need to check what the other apache threads are doing. You might already know this, but if there are three sites, one of the other three might be consuming all available threads AND not letting them go. Maybe the PHP thread isn't letting go of a MySQL connection, or you keep alives are really long, or something.

What I'd like to see is the start configs for StartServers and MinSpareServers and MaxThreads (are these still true for 2.x? I forget) and your configuration for child processes, if you're running in MPM mode, and not prefork mode.

-george