Welcome to WebmasterWorld Guest from 54.146.217.179

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

My Apache Webserver just broke! Long wait times

     
9:47 am on Jul 18, 2012 (gmt 0)

New User

joined:July 18, 2012
posts: 5
votes: 0


My server has been running Win7 with Apache 2.2.4 for a while, all was well. Tonight it started taking forever to load a page if it ever does. Looking at the Firebug response, it will "Wait" for tens of seconds up to minutes between file gets and it makes it unbearable.

It was working fine for over a year. My network is good. This happens when going locally as well, local to the machine and internal network. Pinging the network is fine too.

I don't really know where to look since nothing changed. I looked at my PHP log and there were no problems. I looked at my apache error logs and it had a couple of thread related errors, but I have little actual traffic right now.

Any tips? Essentially it is offline because nobody will wait 5 minutes for a page to load.
10:38 am on July 18, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 12, 2006
posts:2493
votes: 22


memory issues, maybe? have a look and see how much is being used
4:39 pm on July 18, 2012 (gmt 0)

New User

joined:July 18, 2012
posts: 5
votes: 0


Memory is fine. I have also rebooted the computer a few times. I have many GB's of RAM and it is a heavy multi-core system. The Apache process is barely using any CPU at all like in other posts I have seen with the issue.

My other server services are working fine (remote desktop, ftp, filesharing, media streaming, etc). Just Apache is not serving web pages in anything close to a timely manner.
4:51 pm on July 18, 2012 (gmt 0)

New User

joined:July 18, 2012
posts: 5
votes: 0


This is showing a 250KB page taking 18 minutes to serve... The actual file transfers are fast. This is from the server itself (so local). But the waits are forever.

Image: [s11.postimage.org ]
5:46 pm on July 18, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 12, 2006
posts:2493
votes: 22


the other thing i would think of is DNS issues
6:35 pm on July 18, 2012 (gmt 0)

New User

joined:July 18, 2012
posts: 5
votes: 0


The DNS's are good too. The DNS lookup part of the requests take less then a millisecond.

I believe I am actually "under attack" but not sure what to do about it yet. Looking at the active TCP connections I see many hundreds of connections. Generally there are 5 to 10 from the same IP requesting port N, N+1, N+2, N+3, N+4, etc. Then after those timeout, the same IP is back requesting N+5, N+6, N+7, N+8, N+9, etc. This is coming from many IPs, not just one.

I think because it is so many IPs and they are scanning slowly but in numbers, my Intrusion Detection Policy on my Router is not picking it up... I have never had to deal with it before.
7:12 pm on July 18, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 12, 2006
posts:2493
votes: 22


what about your database processes. maybe they are taking ages for some reason, and eventually timing out. one IP could generate a load of those on every page, depending on how you've written your scripts.
if you've got your databases on another server then a reboot might not have fixed anything
7:24 pm on July 18, 2012 (gmt 0)

Junior Member

5+ Year Member

joined:June 4, 2008
posts: 195
votes: 0


Just throwing this out there because I had similar symptoms one time without any errors showing up - check to see if there is an excessively large log file somewhere. I had a log file that wasn't rotating properly and exceeded the server file size limit. The pages on the server would load but it would take forever.
2:31 am on July 19, 2012 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10544
votes: 8


you should whitelist only the ports you need for your applications at the firewall.
thus these connection attempts will fail and your server will never see those requests.
4:50 pm on July 19, 2012 (gmt 0)

New User

joined:July 18, 2012
posts: 5
votes: 0


Unfortunately I have services that need many ports open, ideally all ports. So this will continue to happen.

I would be less concerned and try to block ports at the server if apache was eating all the resources, but it is not. It is using 207 threads and 0.1% average CPU load with 150MB of RAM (out of 16GB reserved for web-stuff). So there is a bottleneck somewhere but I don't know where.