Apache - Overloaded (sites load slowly)

[DarkEden Genesis]

Hello everybody,

I'm experiencing troubles with Apache.

It's overloaded and it results on pages loading slowly.


Server Status (from PLESK)

Services **Needs attention**
Apache CPU usage 24,7% *******
Mail server CPU usage 0%
MySQL CPU usage 1%
Panel CPU usage 0.8%
Apache memory usage 18.1% used (2.79 GB of 15.4 GB)
Mail server memory usage 0% used (6.02 MB of 15.4 GB)
MySQL memory usage 1% used (152 MB of 15.4 GB)
Panel memory usage 0.3% used (52.1 MB of 15.4 GB)

When some service reachs 20% of resource usage, it warns that it's overloaded and that service becomes slow.

At this specific time Apache CPU Usage is the one overloaded.
Apache CPU Usage and Apache Memory Usage are always overloaded....
What's wrong?

[incrediBILL]

More clues please as that's like asking where the needle is in a haystack.

Have you tried restarting Apache?

Running your own dedicated server, is this for one account, all accounts?

Got any big .htaccess files or http.conf files?

Running any bot blockers?

Any high volume websites?

What kind of software is in the accounts, WordPress, vBulletin?

Make sure you aren't hacked or under attack. Unusual spikes in server performance can indicate a spammer running mass amounts of mail thru the server, or a DDoS attack would also spike performance. Hacker/Spam attacks don't always spike the email CPU usage because it's typically more of a bandwidth issue and depending on the type of attack, it could be primarily via Apache, seen it happen.

[DarkEden Genesis]

Hi Bill,

Thanks for your attention.

1- I have tried to restart Apache many and many times already.
You reminded me about some valuable info.
When I restart Apache, the websites start to load faster again, but after sometime they start to decrease performance till it become terrible slow.

2- What you mean by big .htaccess or httpd.conf?
I Have some .htaccess in some websites yes.

3- I don't know...How can I know if I have bot blockers?

4- I don't think so...I tried to deactivate the websites that I guess were using too many resources, but didn't solved the issue.

5- Have a IP Board.


Another reminder: I have checked Apache error and access log and it seems very weird.
Processing requisitions that seems simply crazy.

[lucy24]

Is this your own server? It seems like you'd know what's on it. Are you hosting people who aren't directly connected with you? How many? Has there been any recent change in the number or type of sites you are hosting?

What do you mean by "tried to deactivate"? Tried and failed, or deactivated and it made no difference? I have to assume that people would object if you tried to pull the plug on their sites. That's why I'm puzzled by some of your answers.

Checking the logs should have been one of the very first things you did. What kinds of things are turning up that you'd call "weird" or "simply crazy"? I assume you mean weirder and crazier than usual. Can you compare your current logs against the logs from before your server started acting up?

[DarkEden Genesis]

It's my own server but I have no idea why it's slow...

4th answer update -

"4- I don't think so...I tried to deactivate the websites that I guess were using too many resources,but didn't made difference."

ERROR_LOG

[Sun Feb 19 23:59:46 2012] [error] [client 64.237.54.179] File does not exist: /var/www/vhosts/default/htdocs/st, referer: http://www.search4autos.com/review-2012-buick-lacrosse/

[Sun Feb 19 23:59:47 2012] [error] [client 61.147.67.228] File does not exist: /var/www/vhosts/default/htdocs/st, referer: http://www.gossipgirlinsider.com/2009/10/gossip-girl-spoilers-chair-far-from-over/#comments

[Sun Feb 19 23:59:46 2012] [error] [client 125.75.232.173] proxy: Error reading from remote server returned by /, referer: http://www.whitehouse.net

[Sun Feb 19 23:59:46 2012] [error] [client 65.19.184.246] proxy: Error reading from remote server returned by /, referer: http://www.yt775.com

[Sun Feb 19 23:59:43 2012] [error] (70007)The timeout specified has expired: proxy: HTTP: attempt to connect to 219.139.240.111:80 (*) failed

[Sun Feb 19 23:59:44 2012] [error] (70007)The timeout specified has expired: proxy: HTTP: attempt to connect to 222.186.32.7:80 (*) failed

[Sun Feb 19 23:59:45 2012] [error] (70007)The timeout specified has expired: proxy: HTTP: attempt to connect to 72.14.209.115:80 (*) failed

[Sun Feb 19 23:59:45 2012] [error] (70007)The timeout specified has expired: proxy: HTTP: attempt to connect to 113.11.204.211:80 (*) failed

[Sun Feb 19 23:59:46 2012] [error] [client 79.123.147.216] proxy: Error reading from remote server returned by /

[Sun Feb 19 23:59:46 2012] [error] [client 79.123.147.216] (104)Connection reset by peer: proxy: error reading status line from remote server 64.12.202.43

ACCESS_LOG

27.156.78.6 - - [20/Feb/2012:00:06:19 +0000] "GET [113404url.displayadfeed.com...] HTTP/1.0" 302 - "http://www.intute.us/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

218.78.213.145 - - [20/Feb/2012:00:06:16 +0000] "GET [playsf.cc...] HTTP/1.1" 200 3878 "http://www.whitehouse.net" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 5.1)"

184.82.73.47 - - [20/Feb/2012:00:06:19 +0000] "GET [iranhack.org...] HTTP/1.1" 302 267 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

189.79.120.179 - - [20/Feb/2012:00:06:19 +0000] "A" 400 307 "-" "-"

27.156.78.6 - - [20/Feb/2012:00:06:19 +0000] "GET [media.imgdefault.com...] HTTP/1.0" 200 - "http://www.intute.us/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"

120.32.57.143 - - [20/Feb/2012:00:06:19 +0000] "GET [popunder.popcde.com...] HTTP/1.0" 200 10528 "http://www.intute.us/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)"

109.162.47.139 - - [20/Feb/2012:00:06:17 +0000] "GET [google.com...] HTTP/1.0" 302 356 "http://www.google.com/search?as_q=inurl:%22forum/post/14.html%22&num=100&hl=en&output=ie&filter=0" "Mozilla/4.0 (compatible
MSIE 6.0; Windows NT 5.1; Deepnet Explorer 1.5.0; .NET CLR 1.0.3705)"

111.194.245.156 - - [20/Feb/2012:00:06:18 +0000] "GET [hits.blog.sina.com.cn...] HTTP/1.0" 200 49 "http://blog.sina.com.cn/s/blog_68c2a7cd0102dse0.html" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0)"

125.65.45.197 - - [20/Feb/2012:00:06:20 +0000] "GET [ad.globe7.com...] HTTP/1.0" 404 263 "http://www.quacast.com/" "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Media Center PC 4.0; SLCC1; .NET CLR 3.0.04320)"

61.164.176.131 - - [20/Feb/2012:00:06:19 +0000] "GET [weibo.cn...] HTTP/1.1" 200 1229 "http://weibo.cn/u/1885103407" "Mitsu/1.2.C (MT330) MMP/1.1"

109.230.246.227 - - [20/Feb/2012:00:06:17 +0000] "GET [173.194.32.49...] HTTP/1.0" 302 355 "http://173.194.32.49/search?as_q=10kg+tumble+dryers+jump+to&num=100&hl=en&output=ie&filter=0" "Opera/9.00 (Windows NT 4.0; U; en)"

98.126.134.114 - - [20/Feb/2012:00:06:20 +0000] "GET [r8r88.com...] HTTP/1.1" 302 143 "http://r8r88.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 5.1)"

128.196.239.73 - - [20/Feb/2012:00:06:20 +0000] "GET [images.google.com...] HTTP/1.1" 200 32077 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; .NET CLR 1.1.4322; Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E)"

65.19.184.246 - - [20/Feb/2012:00:04:20 +0000] "GET [yt775.com...] HTTP/1.1" 503 276 "http://www.yt775.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 5.1)"

173.231.55.45 - - [20/Feb/2012:00:04:20 +0000] "GET [123shaiya.com...] HTTP/1.1" 503 279 "http://www.123shaiya.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 5.1)"
84.22.159.150 - - [20/Feb/2012:00:06:20 +0000] "GET [market.yandex.ru...] HTTP/1.1" 302 - "-" "Opera/9.64 (Windows NT 5.1; U; ru) Presto/2.1.1"

59.56.111.54 - - [20/Feb/2012:00:06:42 +0000] "GET [kojiki.server.ne.jp...] HTTP/1.0" 200 2629 "-" ""

199.192.155.176 - - [20/Feb/2012:00:06:43 +0000] "GET [26316.com...] HTTP/1.1" 503 47 "http://www.whitehouse.net" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 5.1)"

123.53.200.171 - - [20/Feb/2012:00:06:42 +0000] "GET [mm.taobao.com...] HTTP/1.0" 200 9784 "http://mm.taobao.com/176817195.htm" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98)"

98.126.134.114 - - [20/Feb/2012:00:06:43 +0000] "GET [r8r88.com...] HTTP/1.1" 302 143 "http://r8r88.com" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 5.1)

[edited by: tedster at 4:33 am (utc) on Feb 21, 2012]

[lucy24]

Yeah. "Weird" and "crazy" about sums it up. Are those the real IP numbers in the error logs or did you make them up?

attempt to connect to 72.14.209.115:80 (*) failed

That's google Preview and Wireless Transcoder-- here I'd put my money on a robot using Wireless Transcoder as a proxy. (They also use Translate.)

... [client 79.123.147.216] (104)Connection reset by peer: proxy: error reading status line from remote server 64.12.202.43

64.12.etc. is AOL. What's their connection to any of this?

What's inside all those GET statements? They come through in your post as stuff in brackets that can't possibly have landed on your server in the form I'm seeing. Unless the problem has nothing to do with your server and there's a huge DNS mess somewhere upstream-- and if so, we'd have heard about it!

Is your server responsible for all those 200s and 302s for people who should never have gotten inside the door? (I noticed especially the blank UA, and a really amazing number of robots in MSIE 6 costumes.) If it were in good health, what kinds of responses would all those requests be getting? The only ones that look right are the 400s and 404s. Can't say about the 503s from Access Logs alone.

It might have been helpful to show access logs and error logs from the same time period so you can see what's going on from both sides. Often you need to fit both sides together to get a full picture. Like puzzle pieces.

[DarkEden Genesis]

Those are all the real numbers...copy and past.

I didn't get the "Transcoder Thing". Can explain easier?

AOL...I don't know what is the connection of that.

I'm sorry, but both my english and programming skills aren't good.
I'm reading many times your post to understand it more and more.

I see too many external addresses, like they were running from my server.
I wonder why so many external addresses...

I think there is something wrong with my Apache.
In 10 minutes of service, Apache processed:
2.5MB of access_log
0.5MB of error_log

Isn't it too large logs for ONLY TEN MINUTES of service?


Can you tell me in easier words what you suspect that is happening on my server?


Thank you again Lucy

[lucy24]

Can you tell me in easier words what you suspect that is happening on my server?

I wish someone would tell me in easier words! In fact, if someone explains it in hard words I can probably translate to easy words, but I can't do both :(

In 10 minutes of service, Apache processed:
2.5MB of access_log
0.5MB of error_log
Isn't it too large logs for ONLY TEN MINUTES of service?

It depends on your site. On my site, that would be several weeks of logs ;) On Google it would probably take about ten seconds.

But your logs are weird. They look as if people are trying to use your server to get to other places.

Are there websites on your server that belong to other people? Or are they all your own sites?

[phranque]

regarding the proxy problem - if i had to guess that is what is slowing your server response.
what version of apache are you using?

regarding the external addresses - it's possible someone is targeting your server for referrer spam.

[DarkEden Genesis]

Hello, thanks for answers.
A friend of mine helped me to find and fix the issues.

Three mistakes:

1) permissions for "/" was for a ftp user (not for administrator). Some bad use of "chown".
Since the default website turned unreadable, all requests was being processed by a vhost, opening a proxy.

2) wrong permissions for "/tmp"

3) Unnecessary and disabled by default modules were loading on apache