Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

From: Donald Buczek
Date: Sun Nov 27 2016 - 04:19:37 EST


On 24.11.2016 11:15, Michal Hocko wrote:
On Mon 21-11-16 16:35:53, Donald Buczek wrote:
[...]
Hello,

thanks a lot for looking into this!

Let me add some information from the reporting site:

* We've tried the patch from Paul E. McKenney (the one posted Wed, 16 Nov
2016) and it doesn't shut up the rcu stall warnings.

* Log file from a boot with the patch applied ( grep kernel
/var/log/messages ) is here :
http://owww.molgen.mpg.de/~buczek/321322/2016-11-21_syslog.txt

* This system is a backup server and walks over thousands of files sometimes
with multiple parallel rsync processes.

* No rcu_* warnings on that machine with 4.7.2, but with 4.8.4 , 4.8.6 ,
4.8.8 and now 4.9.0-rc5+Pauls patch
I assume you haven't tried the Linus 4.8 kernel without any further
stable patches? Just to be sure we are not talking about some later
regression which found its way to the stable tree.

We've tried v4.8 and got the first rcu stall warnings with this, too. First one after about 20 hours uptime.


* When the backups are actually happening there might be relevant memory
pressure from inode cache and the rsync processes. We saw the oom-killer
kick in on another machine with same hardware and similar (a bit higher)
workload. This other machine also shows a lot of rcu stall warnings since
4.8.4.

* We see "rcu_sched detected stalls" also on some other machines since we
switched to 4.8 but not as frequently as on the two backup servers. Usually
there's "shrink_node" and "kswapd" on the top of the stack. Often
"xfs_reclaim_inodes" variants on top of that.
I would be interested to see some reclaim tracepoints enabled. Could you
try that out? At least mm_shrink_slab_{start,end} and
mm_vmscan_lru_shrink_inactive. This should tell us more about how the
reclaim behaved.

http://owww.molgen.mpg.de/~buczek/321322/2016-11-26.dmesg.txt (80K)
http://owww.molgen.mpg.de/~buczek/321322/2016-11-26.trace.txt (50M)

Traces wrapped, but the last event is covered. all vmscan events were enabled


--
Donald Buczek
buczek@xxxxxxxxxxxxx
Tel: +49 30 8413 1433