Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

From: Christopher S. Aker
Date: Sun Nov 27 2016 - 00:42:33 EST



> On Nov 24, 2016, at 5:15 AM, Michal Hocko <mhocko@xxxxxxxxxx> wrote:
>
>> * No rcu_* warnings on that machine with 4.7.2, but with 4.8.4 , 4.8.6 ,
>> 4.8.8 and now 4.9.0-rc5+Pauls patch
>
> I assume you haven't tried the Linus 4.8 kernel without any further
> stable patches? Just to be sure we are not talking about some later
> regression which found its way to the stable tree.

We are also seeing this frequently on our fleet since moving from 4.7.x to 4.8. This is from a machine running vanilla 4.8.6 just a few moments ago:

INFO: rcu_sched detected stalls on CPUs/tasks:
13-...: (420 ticks this GP) idle=ce1/140000000000000/0 softirq=225550784/225550904 fqs=87105
(detected by 26, t=600030 jiffies, g=68185325, c=68185324, q=344996)
Task dump for CPU 13:
kswapd1 R running task 12200 1840 2 0x00000808
0000000000000001 0000000000000034 000000000000012b 0000000000003139
ffff8b643fffb000 ffff8b028cee7cf8 ffff8b028cee7cf8 ffff8b028cee7d08
ffff8b028cee7d08 ffff8b028cee7d18 ffff8b028cee7d18 ffff8b0200000000
Call Trace:
[] ? shrink_node+0xcd/0x2f0
[] ? kswapd+0x304/0x710
[] ? mem_cgroup_shrink_node+0x160/0x160
[] ? kthread+0xc4/0xe0
[] ? ret_from_fork+0x1f/0x40
[] ? kthread_worker_fn+0x140/0x140

The machine will lag terribly during these occurrences .. some will eventually recover, some will spiral down and require a reboot.

-Chris