Apache 2 static file serving performance

What performance can be expected serving lots of small images?

[count_zer0]

I am running an Apache 2 webserver on a Pentium 4 2.8GHz with 2GB RAM. The function of the webserver is as a tag server i.e. it is serving a 1x1 pixel GIF and logging these requests. The size of the image is 43 bytes, the total size of a request is typically 500 bytes. The machine is serving 1 million requests per day, with approximately 30 per second at peak times.

I am have performance monitoring software checking the request times, and during peak times (lunchtime, early evenings), these are very slow - between 2 and 20 seconds to serve this GIF.

1. Based on the above spec. machine, is this a poor performance? I thought Apache was capable of much higher performance.

2. I am an Apache newbie - how do I go about identifying the bottleneck? I have looked in the error.log and there are no load-related messages.

3. Is there a way of tweaking the Apache settings to improve the response times? I would be grateful for any suggestions for specific settings to tweak rather than a link to the performance tuning notes. My current performance settings in httpd.conf are:

# http network level performance settings
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15

Any help would be greatly appreciated.

[count_zer0]

Here is the result of running Apache Benchmark. ab was run on another machine at the same ISP.

ab -n 1000 -c 100 [servertotest.com...]

Concurrency Level: 100
Time taken for tests: 0.230186 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 588174 bytes
HTML transferred: 43086 bytes
Requests per second: 4344.31 [#/sec] (mean)
Time per request: 23.019 [ms] (mean)
Time per request: 0.230 [ms] (mean, across all concurrent requests)
Transfer rate: 2493.64 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.7 0 6
Processing: 5 20 6.4 21 192
Waiting: 4 19 6.4 20 191
Total: 11 20 5.9 21 192

Percentage of the requests served within a certain time (ms)
50% 21
66% 21
75% 21
80% 21
90% 22
95% 22
98% 23
99% 23
100% 192 (longest request)

So the machine seems very capable of handling much higher request rates. Why then are the response times so slow at peak times? Is this an ISP / network thing?

[Frank_Rizzo]

You may want to check the apache cache mem settings. I think the default is to not cache files smaller than nnn bytes. You may need to adjust this so that the gif is always served from cache.

The other thing could be your log file. The more requests you get the more apache has to write to the log file. This can be a bottleneck as the disk has to constantly read the requested files and then write that it served them.

Either turn off logging or try and have your log files saved to a second disk on a different channel.

[count_zer0]

Thanks for your suggestions, Frank.

I will play with the memory settings for Apache.

I don't think that logging is the issue as the ab benchmark process requested the image 1000 times, with 100 concurrent requests, all of which were served and logged in 129ms. In normal operation, the request load is 30 per second.

I'm beginning to suspect that the ISP is having network problems at peak times...

[count_zer0]

I have solved the problem by enabling mod_status. This identified a lot of keep-alives and processes left by gracefully restarting. These were hogging the processes and causing new requests to be queued for ages. Dropping the KeepAliveTimeout setting to 5 seconds, increasing the MaxClients and restarting Apache has fixed the problem.