Welcome to WebmasterWorld Guest from 50.19.53.104

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

My Apache Webserver just broke! Long wait times

     

NVergunst

9:47 am on Jul 18, 2012 (gmt 0)



My server has been running Win7 with Apache 2.2.4 for a while, all was well. Tonight it started taking forever to load a page if it ever does. Looking at the Firebug response, it will "Wait" for tens of seconds up to minutes between file gets and it makes it unbearable.

It was working fine for over a year. My network is good. This happens when going locally as well, local to the machine and internal network. Pinging the network is fine too.

I don't really know where to look since nothing changed. I looked at my PHP log and there were no problems. I looked at my apache error logs and it had a couple of thread related errors, but I have little actual traffic right now.

Any tips? Essentially it is offline because nobody will wait 5 minutes for a page to load.

londrum

10:38 am on Jul 18, 2012 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



memory issues, maybe? have a look and see how much is being used

NVergunst

4:39 pm on Jul 18, 2012 (gmt 0)



Memory is fine. I have also rebooted the computer a few times. I have many GB's of RAM and it is a heavy multi-core system. The Apache process is barely using any CPU at all like in other posts I have seen with the issue.

My other server services are working fine (remote desktop, ftp, filesharing, media streaming, etc). Just Apache is not serving web pages in anything close to a timely manner.

NVergunst

4:51 pm on Jul 18, 2012 (gmt 0)



This is showing a 250KB page taking 18 minutes to serve... The actual file transfers are fast. This is from the server itself (so local). But the waits are forever.

Image: [s11.postimage.org ]

londrum

5:46 pm on Jul 18, 2012 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



the other thing i would think of is DNS issues

NVergunst

6:35 pm on Jul 18, 2012 (gmt 0)



The DNS's are good too. The DNS lookup part of the requests take less then a millisecond.

I believe I am actually "under attack" but not sure what to do about it yet. Looking at the active TCP connections I see many hundreds of connections. Generally there are 5 to 10 from the same IP requesting port N, N+1, N+2, N+3, N+4, etc. Then after those timeout, the same IP is back requesting N+5, N+6, N+7, N+8, N+9, etc. This is coming from many IPs, not just one.

I think because it is so many IPs and they are scanning slowly but in numbers, my Intrusion Detection Policy on my Router is not picking it up... I have never had to deal with it before.

londrum

7:12 pm on Jul 18, 2012 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



what about your database processes. maybe they are taking ages for some reason, and eventually timing out. one IP could generate a load of those on every page, depending on how you've written your scripts.
if you've got your databases on another server then a reboot might not have fixed anything

getcooking

7:24 pm on Jul 18, 2012 (gmt 0)

5+ Year Member



Just throwing this out there because I had similar symptoms one time without any errors showing up - check to see if there is an excessively large log file somewhere. I had a log file that wasn't rotating properly and exceeded the server file size limit. The pages on the server would load but it would take forever.

phranque

2:31 am on Jul 19, 2012 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



you should whitelist only the ports you need for your applications at the firewall.
thus these connection attempts will fail and your server will never see those requests.

NVergunst

4:50 pm on Jul 19, 2012 (gmt 0)



Unfortunately I have services that need many ports open, ideally all ports. So this will continue to happen.

I would be less concerned and try to block ports at the server if apache was eating all the resources, but it is not. It is using 207 threads and 0.1% average CPU load with 150MB of RAM (out of 16GB reserved for web-stuff). So there is a bottleneck somewhere but I don't know where.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month