homepage Welcome to WebmasterWorld Guest from 54.198.42.105
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
Forum Library, Charter, Moderators: bakedjake

Linux, Unix, and *nix like Operating Systems Forum

    
High loading but Low CPU usage
foxfox




msg:3339314
 4:32 pm on May 14, 2007 (gmt 0)

I have some problems in my current servers setup, hope someone can help....

2 servers:

Server A: NFS server, for serving static HTML publishing (frequently changed, real-time, can't be rsync-ed)
Server B: Web server, for serving data stored in Server A, and send to client

in the future, we might add C, D, E, ... and mount to Server A (via. NFS)

One of the problem with this setup is Server B is having high system loading when traffic is high (but cpu usage is not that high), e.g.
================================================================================

top - 23:42:26 up 113 days, 21:37, 2 users, load average: 5.50, 7.15, 5.82
Tasks: 452 total, 3 running, 449 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.2% us, 3.6% sy, 0.0% ni, 84.1% id, 3.1% wa, 0.0% hi, 0.0% si
Mem: 4149144k total, 4024796k used, 124348k free, 23424k buffers
Swap: 15358100k total, 516k used, 15357584k free, 1759136k cached

and iostat,

avg-cpu: %user %nice %sys %iowait %idle
9.17 0.00 3.62 3.07 84.14

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 42.64 188.30 116.79 1844743172 1144159376
sda1 16.45 0.58 131.36 5720906 1286897928
sda2 0.00 0.35 0.00 3399990 560
sda3 24.38 155.18 53.21 1520305323 521331664
sda5 48.51 31.62 366.88 309817218 3594202728
sda6 0.00 0.00 0.01 24825 109104
sda7 0.48 0.56 3.73 5455430 36572328

================================================================================
for Server A (NFS server), CPU loading is:

23:45:11 up 448 days, 23:38, 1 user, load average: 3.81, 5.46, 5.50
155 processes: 152 sleeping, 2 running, 1 zombie, 0 stopped
CPU0 states: 13.3% user 5.2% system 0.0% nice 0.0% iowait 80.3% idle
CPU1 states: 9.0% user 1.2% system 0.0% nice 0.0% iowait 89.1% idle
CPU2 states: 13.2% user 3.2% system 0.0% nice 0.0% iowait 82.4% idle
CPU3 states: 11.3% user 2.2% system 0.0% nice 0.0% iowait 85.4% idle
Mem: 3355136k av, 2991820k used, 363316k free, 0k shrd, 111356k buff
884720k active, 659996k inactive
Swap: 8184188k av, 236320k used, 7947868k free 1288932k cached

and iostat,

avg-cpu: %user %nice %sys %idle
66.77 0.03 20.18 13.02

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 607.68 94.88 113.66 624284376 747916866
dev8-1 28.94 230.73 182.09 1518207266 1198153062

also, /proc/net/rpc/nfsd

================================================================================

any recommendation/comments can help in the above setup?

Thanks...

 

foxfox




msg:3339315
 4:33 pm on May 14, 2007 (gmt 0)

for the NFS server (Server A)

cat /proc/net/rpc/nfsd
rc 2126116 488145714 1889205217
fh 288951 2463282720 0 135034 364483
io 2170871665 1260986404
th 8 152384263 957475.460 223366.410 843524.850 0.000 671911.400 520621.870 399458.440 304512.060 0.000 830938.230
ra 16 132540471 407091 72150 88010 34713 54685 46379 22455 47001 24140 89626751
net -1915490134 852897161 1526580001 27092
rpc -1915490249 115 115 0 0
proc2 18 139 866768420 74333853 0 799076471 1 222964439 0 243457662 58134137 56119428 53172728 0 0 119968 3 4934051 395747

physics




msg:3339391
 6:10 pm on May 14, 2007 (gmt 0)

Did you try running netstat on both machines?

I'm not an expert on this but the setup seems a bit off to me. I.e. there is no redundancy if something happens to server A and A will always bee a bottleneck. It may be better to have identical servers and do load balancing. That way there's redundancy and you shouldn't run into these network/disk bottlenecks (I'm guessing that is in fact the problem). Also, it'll be easier to deal with load issues ... just drop another identical server in.

mcavic




msg:3340488
 10:38 pm on May 15, 2007 (gmt 0)

A high load average with low CPU usage usually means that the drives are too slow. In this case, it might be the NFS.

How fast is the connection between the two servers? If you're going to continue using NFS, they should be using gigabit Ethernet, with a gigabit switch.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Linux, Unix, and *nix like Operating Systems
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved