Re: extreme system load [kswapd]

[Nicolas Maupu]

Hello,

Maybe, try to track IO on your machine (iostat) to see if it is an IO
problem (i.e. write to disk) or not.


On Tue, Mar 20, 2012 at 10:08, Karol Åebesta <sebesta.karol@xxxxxxxxx> wrote:
>
> Hello @All,
>
> We have a problem on our production machine with high CPU utilization
> caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
> of SWAP.
>
>
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.6 (Tikanga)
> #
>
>
> # uptime
> Â09:52:13 up 38 days, 17:01, 14 users, Âload average: 43.93, 50.02, 51.36
> #
>
>
> # free -m
>       total    used    free   shared  Âbuffers   cached
> Mem: Â Â Â Â128989 Â Â Â75577 Â Â Â53412 Â Â Â Â Â0 Â Â Â Â416 Â Â Â57131
> -/+ buffers/cache: Â Â Â18029 Â Â 110960
> Swap: Â Â Â Â81919 Â Â Â31310 Â Â Â50609
> #
>
> ### from TOP:
>
> Tasks: 1590 total, Â 3 running, 1578 sleeping, Â 1 stopped, Â 8 zombie
> Cpu(s): Â1.1%us, Â1.9%sy, Â0.0%ni, 74.4%id, 22.4%wa, Â0.0%hi, Â0.2%si, Â0.0%st
> Mem: Â132085528k total, 77397544k used, 54687984k free, Â 426472k buffers
> Swap: 83886064k total, 32043168k used, 51842896k free, 58475692k cached
>
> ÂPID USER Â Â ÂPR ÂNI ÂVIRT ÂRES ÂSHR S %CPU %MEM Â ÂTIME+ ÂCOMMAND
> Â1816 root   Â10 Â-5   0  Â0  Â0 D 203.0 Â0.0  1124:39 kswapd3
> 10801 root   Â15  0 Â113m Â23m 1680 S 158.6 Â0.0  1694:13 vxconfigd
> Â3925 root   ÂRT  0 Â162m Â15m 3368 S 140.1 Â0.0  1313:32 multipathd
> 21399 wsy5067 Â 16 Â 0 22768 3192 1704 S 50.3 Â0.0 Â 0:12.96 top
> 10583 root   Â15  0   0  Â0  Â0 S 43.8 Â0.0 Â58:56.55 dmp_daemon
> 16673 root   Â16  0 17720 1220 Â792 S 37.3 Â0.0 732:23.49 cmafcad
> 24636 oracle  Â15  0 3314m Â41m Â40m S 26.6 Â0.0 Â29:31.84 oracle
> Â487 oracle  Â15  0 3301m Â90m Â88m S 21.4 Â0.1  1:05.68 oracle
> 14197 root   Â18  0 Â315m Â15m 3984 S 21.1 Â0.0 346:15.70 vxsvc
> 26071 root   Â15  0 23900 6788 2092 S 19.8 Â0.0 Â12:31.44 MountAgent
> 16418 root   Â15  0 Â5308 3656 Â592 S 19.1 Â0.0 544:54.78 cmahostd
> Â485 oracle  Â15  0 3301m Â87m Â87m S 18.8 Â0.1  1:03.92 oracle
> 21456 root   Â15  0   0  Â0  Â0 S 18.5 Â0.0 Â14:16.09 pdflush
> Â4699 root   Â22  0 65204 8420 1568 S 15.6 Â0.0 137:15.75 vxdclid
> 16603 root   Â18  0 Â8820 6808 Â332 D 14.3 Â0.0  1290:32 cmaperfd
> 22047 tlmcron1 Â18 Â 0 1227m 115m 6812 S 12.3 Â0.1 554:55.86 java
> Â6157 oracle  Â16  0 3302m 118m 115m D 11.4 Â0.1  0:01.29 oracle
> 30669 oracle  Â16  0 2066m 141m 134m D 10.7 Â0.1  0:20.19 oracle
> Â357 tlmtst3 Â 18 Â 0 Â101m Â15m 9292 S Â9.7 Â0.0 Â 0:29.43 stlfetch
> 15005 oracle  Â16  0 2046m Â84m Â75m D Â9.7 Â0.1  0:01.75 oracle
> 32744 tlmtst3 Â 18 Â 0 Â101m Â15m 9292 S Â8.8 Â0.0 Â 0:31.52 stlfetch
> 26077 root   Â15  0 12116 4228 1936 S Â7.8 Â0.0  4:26.71 VolumeAgent
> 17955 oracle  Â16  0 3301m Â28m Â25m D Â7.1 Â0.0  0:00.22 oracle
> 26076 root   Â15  0 23268 5284 2224 S Â6.8 Â0.0  4:00.50 OracleAgent
> Â111 root   Â10 Â-5   0  Â0  Â0 S Â6.5 Â0.0  4:57.32 events/13
> Â5112 oracle  Â16  0 3314m Â35m Â35m S Â6.5 Â0.0 Â10:38.55 oracle
> 10440 oracle  Â16  0 3314m Â36m Â36m S Â6.5 Â0.0 Â10:54.65 oracle
> 26074 root   Â15  0 23012 4904 2136 S Â6.5 Â0.0  3:09.46 NetlsnrAgent
> Â8631 oracle  Â16  0 3315m Â37m Â36m S Â6.2 Â0.0 Â12:04.57 oracle
> 26636 oracle  Â15  0 3316m Â40m Â39m S Â6.2 Â0.0 Â13:07.75 oracle
> 10780 oracle  Â16  0 6386m Â39m Â39m D Â5.8 Â0.0 Â10:57.01 oracle
> 13091 tlmtst5 Â 15 Â 0 Â423m Â18m Â13m S Â5.8 Â0.0 Â 0:02.76 gedit
> 17212 oracle  Â15  0 2050m Â39m Â39m S Â5.8 Â0.0 Â11:30.82 oracle
> 20283 oracle  Â15  0 3314m Â41m Â40m S Â5.8 Â0.0 Â10:49.00 oracle
> 13147 oracle  Â15  0 3314m Â43m Â43m S Â5.5 Â0.0 Â10:43.12 oracle
> 16875 oracle  Â15  0 3316m Â42m Â40m S Â5.5 Â0.0 Â22:52.66 oracle
> 28143 root   Â15  0 Â111m 6700 3396 S Â5.5 Â0.0 143:25.46 opcmona
> Â5718 oracle  Â16  0 3314m Â38m Â36m S Â5.2 Â0.0 Â11:30.95 oracle
> 22163 oracle  Â15  0 3316m Â39m Â38m S Â5.2 Â0.0 Â32:51.94 oracle
> 26065 root   Â15  0 22960 4724 2164 S Â4.5 Â0.0  1:53.23 ApplicationAgen
> 26612 oracle  Â18  0 3302m Â22m Â19m S Â4.2 Â0.0 Â23:31.48 oracle
> Â8597 oracle  Â18  0 3302m Â18m Â16m S Â3.9 Â0.0 Â23:30.62 oracle
> 10690 oracle  Â18  0 6373m Â20m Â16m S Â3.9 Â0.0 Â23:28.74 oracle
> 11822 oracle  Â18  0 2038m Â29m Â25m S Â3.9 Â0.0  0:05.17 oracle
>
> Why the kswapd daemon causing so high CPU utilization? Can it be
> caused by some process which try to access memory pages which are not
> in memory, but in swap? Or can it be cause by swapping for exapmple
> some inactive oracle database SGA from memory to swap?
>
> Thank you for help.
>
>
> Regards,
>
> Karol Sebesta
> --
> To unsubscribe from this list: send the line "unsubscribe linux-admin" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at Âhttp://vger.kernel.org/majordomo-info.html




--

Nicolas Maupu - SS2J - Excilys
TÃl :+33 (0) 1 41 24 43 27
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/