Re: [PATCH] mm/kmemleak: report RCU-tasks quiescent states during the scan

[Paul E. McKenney]

On Fri, Jul 24, 2026 at 02:58:24AM -0700, Breno Leitao wrote:
> On Thu, Jul 23, 2026 at 08:56:58AM -0700, Paul E. McKenney wrote:
> > On Thu, Jul 23, 2026 at 06:26:11AM -0700, Breno Leitao wrote:
> > > Hello Paul,
> > > 
> > > On Mon, Jul 20, 2026 at 09:21:17PM -0700, Paul E. McKenney wrote:
> > > > On Mon, Jul 20, 2026 at 03:39:17PM -0700, Andrew Morton wrote:
> > > > > On Mon, 20 Jul 2026 06:23:45 -0700 Breno Leitao <leitao@xxxxxxxxxx> wrote:
> > > > > 
> > > > > > kmemleak_scan() can run for ages on large debug kernels. It was
> > > > > > causing some soft-lockups which I got fixed with commit
> > > > > > 3175fcfec8b16baeb ("mm/kmemleak: avoid soft lockup when scanning task
> > > > > > stacks") with our beloved cond_resched().
> > > > > > 
> > > > > > I've got the fix above deployed in the Meta fleet, and now I am seeing:
> > > > > > 
> > > > > >   INFO: rcu_tasks detected stalls on tasks:
> > > > > >     task:kmemleak state:R ... nvcsw: 274/274 holdout: 1 idle_cpu: -1/3
> > > > > >      scan_block
> > > > > >      scan_gray_list
> > > > > >      kmemleak_scan
> > > > > > 
> > > > > > and, worse, blocks the callers waiting on that grace period. Here a BPF
> > > > > > struct_ops map free, which waits via synchronize_rcu_mult(call_rcu,
> > > > > > call_rcu_tasks), is stuck long enough to also trip the hung task check:
> > > > > > 
> > > > > >   INFO: task kworker/...:bpf_map_free_deferred blocked for 122 seconds
> > > > > >      __wait_rcu_gp
> > > > > >      bpf_struct_ops_map_free
> > > > > > 
> > > > > > Then I've learned that cond_resched() is not an RCU-tasks quiescent
> > > > > > state, so, we need to use stronger primitives.
> > > > > > 
> > > > > > Use cond_resched_tasks_rcu_qs() at the scan reschedule points so the scan
> > > > > > reports an RCU-tasks quiescent state as it proceeds.
> > > > > > 
> > > > > > Inspired by commit b96285e10aad ("tracing: Have osnoise_main() add a
> > > > > > quiescent state for task rcu").
> > > > > 
> > > > > I'll add
> > > > > 
> > > > > Fixes: c4b28963fd79 ("mm/kmemleak: rely on rcu for task stack scanning")
> > > > > Cc: <stable@xxxxxxxxxxxxxxx>
> > > > G
> > > > Thanks to all three of you!
> > > > 
> > > > This adds fewer than ten calls to cond_resched_tasks_rcu_qs(), but still
> > > > more than doubles the number of such calls outside of the RCU subsystem.
> > > > 
> > > > Which is most likely just fine, and in any case absolutely should not
> > > > get in the way of Breno's patch, which after all solves a real problem
> > > > in the here and now.
> > > > 
> > > > Nevertheless, on the off-chance that over the next few months or years
> > > > we start playing cond_resched_tasks_rcu_qs() whack-a-mole, I figured it
> > > > would be good to get a head start on writing up alternatives.  An initial
> > > > draft may be found here:
> > > > 
> > > > https://docs.google.com/document/d/1s3fn29SCTYVw9jak4iraNIVQR59Wk5C6_I-_eu-MfQA/edit?usp=sharing
> > > > 
> > > > TL;DR:  Should we get into a rousing game of whack-a-mole, alternatives
> > > > include continuing as we are, making the existing calls to cond_resched()
> > > > in turn call cond_resched_tasks_rcu_qs(), decoupling mutex-induced
> > > > hung-task warnings from synchronize_rcu_tasks(), and various
> > > > not-so-practical alternatives to RCU Tasks for trampoline synchronization.
> > > > 
> > > > Thoughts?  Especially thoughts on other schemes?
> > > 
> > > While debugging this issue, I was surprised to discover that cond_resched()
> > > doesn't provide RCU-tasks quiescent states. That led me to
> > > cond_resched_tasks_rcu_qs(), which is the stronger primitive needed for
> > > long-running kernel threads like kmemleak_scan(). Is this the common
> > > case for cond_resched()? I got the impression that kmemleak is the
> > > extreme side, but, I have no data on this.
> > 
> > And that discovery did in fact lead you correctly, and hence my
> > giving you a Reviewed-by.
> > 
> > > Worth noting that commit 7dadeaa6e851e7 ("sched: Further restrict the
> > > preemption modes") continues to narrow PREEMPT_NONE, so the future of
> > > cond_resched() itself may be uncertain, no?
> > 
> > Especially given that one of the explicitly stated goals of lazy
> > preemption was in fact to eliminate cond_resched().  ;-)
> > 
> > Getting back to your point about kmemleak possibly being extreme, as far
> > as I know, none of us have data on how many conversions of cond_resched()
> > to cond_resched_tasks_rcu_qs() will be required, or, for that matter,
> > how many additions of cond_resched_tasks_rcu_qs() above and beyond
> > current cond_resched() instances will be required.
> 
> I've searched for "rcu_tasks detected stalls on tasks" on Meta's
> fleet "dmesg depot" and I was suprised to see we have many of them, in
> which seems to be on hypervisor workloads (running VMs).

Good to know!  There have been experiments in the past with hinting
to the hypervisor that a given vCPU could really use some CPU time,
but the result has usually instead been a more targeted fix.

> > There were concerns raised off-list, so I figured that it was time
> > to lay out what we do know, hence the above gdoc.  I very much hope
> > that we don't need many more calls to cond_resched_tasks_rcu_qs(), and
> > thus that the time I spent writing that turns out to be wasted time.
> > But better safe than sorry!
> 
> The document was not a waste of time. it helped people like me to
> understand what the issue is, and what are the decisions we have ahead
> of us.
> 
> Back to what is the best decision/design, I honestly don't have an
> opinion, but I am happy to get more data about Meta production servers
> to help with the decision.

Looking forward to seeing what you come up with!

							Thanx, Paul