Re: extreme system load [kswapd]

From: Nicolas Maupu
Date: Tue Mar 20 2012 - 05:47:25 EST


Hello,

Maybe, try to track IO on your machine (iostat) to see if it is an IO
problem (i.e. write to disk) or not.


On Tue, Mar 20, 2012 at 10:08, Karol Åebesta <sebesta.karol@xxxxxxxxx> wrote:
>
> Hello @All,
>
> We have a problem on our production machine with high CPU utilization
> caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
> of SWAP.
>
>
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.6 (Tikanga)
> #
>
>
> # uptime
> Â09:52:13 up 38 days, 17:01, 14 users, Âload average: 43.93, 50.02, 51.36
> #
>
>
> # free -m
>       total    used    free   shared  Âbuffers   cached
> Mem: Â Â Â Â128989 Â Â Â75577 Â Â Â53412 Â Â Â Â Â0 Â Â Â Â416 Â Â Â57131
> -/+ buffers/cache: Â Â Â18029 Â Â 110960
> Swap: Â Â Â Â81919 Â Â Â31310 Â Â Â50609
> #
>
> ### from TOP:
>
> Tasks: 1590 total, Â 3 running, 1578 sleeping, Â 1 stopped, Â 8 zombie
> Cpu(s): Â1.1%us, Â1.9%sy, Â0.0%ni, 74.4%id, 22.4%wa, Â0.0%hi, Â0.2%si, Â0.0%st
> Mem: Â132085528k total, 77397544k used, 54687984k free, Â 426472k buffers
> Swap: 83886064k total, 32043168k used, 51842896k free, 58475692k cached
>
> ÂPID USER Â Â ÂPR ÂNI ÂVIRT ÂRES ÂSHR S %CPU %MEM Â ÂTIME+ ÂCOMMAND
> Â1816 root   Â10 Â-5   0  Â0  Â0 D 203.0 Â0.0  1124:39 kswapd3
> 10801 root   Â15  0 Â113m Â23m 1680 S 158.6 Â0.0  1694:13 vxconfigd
> Â3925 root   ÂRT  0 Â162m Â15m 3368 S 140.1 Â0.0  1313:32 multipathd
> 21399 wsy5067 Â 16 Â 0 22768 3192 1704 S 50.3 Â0.0 Â 0:12.96 top
> 10583 root   Â15  0   0  Â0  Â0 S 43.8 Â0.0 Â58:56.55 dmp_daemon
> 16673 root   Â16  0 17720 1220 Â792 S 37.3 Â0.0 732:23.49 cmafcad
> 24636 oracle  Â15  0 3314m Â41m Â40m S 26.6 Â0.0 Â29:31.84 oracle
> Â487 oracle  Â15  0 3301m Â90m Â88m S 21.4 Â0.1  1:05.68 oracle
> 14197 root   Â18  0 Â315m Â15m 3984 S 21.1 Â0.0 346:15.70 vxsvc
> 26071 root   Â15  0 23900 6788 2092 S 19.8 Â0.0 Â12:31.44 MountAgent
> 16418 root   Â15  0 Â5308 3656 Â592 S 19.1 Â0.0 544:54.78 cmahostd
> Â485 oracle  Â15  0 3301m Â87m Â87m S 18.8 Â0.1  1:03.92 oracle
> 21456 root   Â15  0   0  Â0  Â0 S 18.5 Â0.0 Â14:16.09 pdflush
> Â4699 root   Â22  0 65204 8420 1568 S 15.6 Â0.0 137:15.75 vxdclid
> 16603 root   Â18  0 Â8820 6808 Â332 D 14.3 Â0.0  1290:32 cmaperfd
> 22047 tlmcron1 Â18 Â 0 1227m 115m 6812 S 12.3 Â0.1 554:55.86 java
> Â6157 oracle  Â16  0 3302m 118m 115m D 11.4 Â0.1  0:01.29 oracle
> 30669 oracle  Â16  0 2066m 141m 134m D 10.7 Â0.1  0:20.19 oracle
> Â357 tlmtst3 Â 18 Â 0 Â101m Â15m 9292 S Â9.7 Â0.0 Â 0:29.43 stlfetch
> 15005 oracle  Â16  0 2046m Â84m Â75m D Â9.7 Â0.1  0:01.75 oracle
> 32744 tlmtst3 Â 18 Â 0 Â101m Â15m 9292 S Â8.8 Â0.0 Â 0:31.52 stlfetch
> 26077 root   Â15  0 12116 4228 1936 S Â7.8 Â0.0  4:26.71 VolumeAgent
> 17955 oracle  Â16  0 3301m Â28m Â25m D Â7.1 Â0.0  0:00.22 oracle
> 26076 root   Â15  0 23268 5284 2224 S Â6.8 Â0.0  4:00.50 OracleAgent
> Â111 root   Â10 Â-5   0  Â0  Â0 S Â6.5 Â0.0  4:57.32 events/13
> Â5112 oracle  Â16  0 3314m Â35m Â35m S Â6.5 Â0.0 Â10:38.55 oracle
> 10440 oracle  Â16  0 3314m Â36m Â36m S Â6.5 Â0.0 Â10:54.65 oracle
> 26074 root   Â15  0 23012 4904 2136 S Â6.5 Â0.0  3:09.46 NetlsnrAgent
> Â8631 oracle  Â16  0 3315m Â37m Â36m S Â6.2 Â0.0 Â12:04.57 oracle
> 26636 oracle  Â15  0 3316m Â40m Â39m S Â6.2 Â0.0 Â13:07.75 oracle
> 10780 oracle  Â16  0 6386m Â39m Â39m D Â5.8 Â0.0 Â10:57.01 oracle
> 13091 tlmtst5 Â 15 Â 0 Â423m Â18m Â13m S Â5.8 Â0.0 Â 0:02.76 gedit
> 17212 oracle  Â15  0 2050m Â39m Â39m S Â5.8 Â0.0 Â11:30.82 oracle
> 20283 oracle  Â15  0 3314m Â41m Â40m S Â5.8 Â0.0 Â10:49.00 oracle
> 13147 oracle  Â15  0 3314m Â43m Â43m S Â5.5 Â0.0 Â10:43.12 oracle
> 16875 oracle  Â15  0 3316m Â42m Â40m S Â5.5 Â0.0 Â22:52.66 oracle
> 28143 root   Â15  0 Â111m 6700 3396 S Â5.5 Â0.0 143:25.46 opcmona
> Â5718 oracle  Â16  0 3314m Â38m Â36m S Â5.2 Â0.0 Â11:30.95 oracle
> 22163 oracle  Â15  0 3316m Â39m Â38m S Â5.2 Â0.0 Â32:51.94 oracle
> 26065 root   Â15  0 22960 4724 2164 S Â4.5 Â0.0  1:53.23 ApplicationAgen
> 26612 oracle  Â18  0 3302m Â22m Â19m S Â4.2 Â0.0 Â23:31.48 oracle
> Â8597 oracle  Â18  0 3302m Â18m Â16m S Â3.9 Â0.0 Â23:30.62 oracle
> 10690 oracle  Â18  0 6373m Â20m Â16m S Â3.9 Â0.0 Â23:28.74 oracle
> 11822 oracle  Â18  0 2038m Â29m Â25m S Â3.9 Â0.0  0:05.17 oracle
>
> Why the kswapd daemon causing so high CPU utilization? Can it be
> caused by some process which try to access memory pages which are not
> in memory, but in swap? Or can it be cause by swapping for exapmple
> some inactive oracle database SGA from memory to swap?
>
> Thank you for help.
>
>
> Regards,
>
> Karol Sebesta
> --
> To unsubscribe from this list: send the line "unsubscribe linux-admin" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at Âhttp://vger.kernel.org/majordomo-info.html




--

Nicolas Maupu - SS2J - Excilys
TÃl :+33 (0) 1 41 24 43 27
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/