My Apache Webserver just broke! Long wait times

[NVergunst]

My server has been running Win7 with Apache 2.2.4 for a while, all was well. Tonight it started taking forever to load a page if it ever does. Looking at the Firebug response, it will "Wait" for tens of seconds up to minutes between file gets and it makes it unbearable.

It was working fine for over a year. My network is good. This happens when going locally as well, local to the machine and internal network. Pinging the network is fine too.

I don't really know where to look since nothing changed. I looked at my PHP log and there were no problems. I looked at my apache error logs and it had a couple of thread related errors, but I have little actual traffic right now.

Any tips? Essentially it is offline because nobody will wait 5 minutes for a page to load.

[londrum]

memory issues, maybe? have a look and see how much is being used

[NVergunst]

Memory is fine. I have also rebooted the computer a few times. I have many GB's of RAM and it is a heavy multi-core system. The Apache process is barely using any CPU at all like in other posts I have seen with the issue.

My other server services are working fine (remote desktop, ftp, filesharing, media streaming, etc). Just Apache is not serving web pages in anything close to a timely manner.

[NVergunst]

This is showing a 250KB page taking 18 minutes to serve... The actual file transfers are fast. This is from the server itself (so local). But the waits are forever.

Image: [s11.postimage.org ]

[londrum]

the other thing i would think of is DNS issues

[NVergunst]

The DNS's are good too. The DNS lookup part of the requests take less then a millisecond.

I believe I am actually "under attack" but not sure what to do about it yet. Looking at the active TCP connections I see many hundreds of connections. Generally there are 5 to 10 from the same IP requesting port N, N+1, N+2, N+3, N+4, etc. Then after those timeout, the same IP is back requesting N+5, N+6, N+7, N+8, N+9, etc. This is coming from many IPs, not just one.

I think because it is so many IPs and they are scanning slowly but in numbers, my Intrusion Detection Policy on my Router is not picking it up... I have never had to deal with it before.

[londrum]

what about your database processes. maybe they are taking ages for some reason, and eventually timing out. one IP could generate a load of those on every page, depending on how you've written your scripts.
if you've got your databases on another server then a reboot might not have fixed anything

[getcooking]

Just throwing this out there because I had similar symptoms one time without any errors showing up - check to see if there is an excessively large log file somewhere. I had a log file that wasn't rotating properly and exceeded the server file size limit. The pages on the server would load but it would take forever.

[phranque]

you should whitelist only the ports you need for your applications at the firewall.
thus these connection attempts will fail and your server will never see those requests.

[NVergunst]

Unfortunately I have services that need many ports open, ideally all ports. So this will continue to happen.

I would be less concerned and try to block ports at the server if apache was eating all the resources, but it is not. It is using 207 threads and 0.1% average CPU load with 150MB of RAM (out of 16GB reserved for web-stuff). So there is a bottleneck somewhere but I don't know where.