Re: BUG: workqueue lockup - SRCU schedules work on not-online CPUs during size transition

[Paul E. McKenney]

On Thu, Apr 09, 2026 at 09:03:26PM -0700, Paul E. McKenney wrote:
> On Thu, Apr 09, 2026 at 01:10:14PM -0700, Paul E. McKenney wrote:
> > On Thu, Apr 09, 2026 at 09:15:50PM +0200, Vasily Gorbik wrote:
> 
> [ . . . ]
> 
> > > Yes, tested on s390 LPAR (76 online, 400 possible) as well as
> > > on x86 KVM with --smp 16,maxcpus=255 and CONFIG_NR_CPUS=256
> > > no more workqueue lockup in both cases.
> > > 
> > > Thank you!
> > > 
> > > Tested-by: Vasily Gorbik <gor@xxxxxxxxxxxxx>
> > 
> > Thank you for testing this!
> > 
> > Please see below for an updated patch.  Tejun's patch might obsolete
> > this one, but just in case he balks at SRCU queueing handlers for CPUs
> > that are not even in the cpu_possible_mask.  ;-)
> 
> And because we don't invoke SRCU callbacks on CPUs that are not yet fully
> online, such CPUs had better not invoke call_srcu(), synchronize_srcu(),
> or synchronize_srcu_expedited() on a CPU that is not yet fully online.
> I am therefore adding the warning shown below.
> 
> Better paranoid late than paranoid not at all.  ;-)

Except that to make that check actually work, additional code is needed.
So much so that it is easier to make call_srcu() from not-yet-onlined
CPUs work properly, the trick being to queue the callback onto the boot
CPU's callback queue.  That way, there is no need to invoke callbacks
queued on CPUs that cannot yet run workqueue handlers.

Please see below for the full patch, including refraining from queueing
workqueue handlers on not-yet-online CPUs and diverting SRCU callbacks
from not-yet-fully-online CPUs to the boot CPU's callback queue.

Thoughts?

							Thanx, Paul

------------------------------------------------------------------------

commit ce533a60b2ef29a9b516cc717e77c6b679bc09c0
Author: Paul E. McKenney <paulmck@xxxxxxxxxx>
Date:   Thu Apr 9 11:16:02 2026 -0700

    srcu: Don't queue workqueue handlers to never-online CPUs
    
    While an srcu_struct structure is in the midst of switching from CPU-0
    to all-CPUs state, it can attempt to invoke callbacks for CPUs that
    have never been online.  Worse yet, it can attempt in invoke callbacks
    for CPUs that never will be online due to not being present in the
    cpu_possible_mask.  This can cause hangs on s390, which is not set up to
    deal with workqueue handlers being scheduled on such CPUs.  This commit
    therefore causes Tree SRCU to refrain from queueing workqueue handlers
    on CPUs that have not yet (and might never) come online.
    
    Because callbacks are not invoked on CPUs that have not been
    online, it is an error to invoke call_srcu(), synchronize_srcu(), or
    synchronize_srcu_expedited() on a CPU that is not yet fully online.
    However, it turns out to be less code to redirect the callbacks
    from too-early invocations of call_srcu() than to warn about such
    invocations.  This commit therefore also redirects callbacks queued on
    not-yet-fully-online CPUs to the boot CPU.
    
    Reported-by: Vasily Gorbik <gor@xxxxxxxxxxxxx>
    Signed-off-by: Paul E. McKenney <paulmck@xxxxxxxxxx>
    Tested-by: Vasily Gorbik <gor@xxxxxxxxxxxxx>
    Cc: Tejun Heo <tj@xxxxxxxxxx>

diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 0d01cd8c4b4a7..7c2f7cc131f7a 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -897,11 +897,9 @@ static void srcu_schedule_cbs_snp(struct srcu_struct *ssp, struct srcu_node *snp
 {
 	int cpu;
 
-	for (cpu = snp->grplo; cpu <= snp->grphi; cpu++) {
-		if (!(mask & (1UL << (cpu - snp->grplo))))
-			continue;
-		srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, cpu), delay);
-	}
+	for (cpu = snp->grplo; cpu <= snp->grphi; cpu++)
+		if ((mask & (1UL << (cpu - snp->grplo))) && rcu_cpu_beenfullyonline(cpu))
+			srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, cpu), delay);
 }
 
 /*
@@ -1322,7 +1320,9 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,
 	 */
 	idx = __srcu_read_lock_nmisafe(ssp);
 	ss_state = smp_load_acquire(&ssp->srcu_sup->srcu_size_state);
-	if (ss_state < SRCU_SIZE_WAIT_CALL)
+	// If !rcu_cpu_beenfullyonline(), interrupts are still disabled,
+	// so no migration is possible in either direction from this CPU.
+	if (ss_state < SRCU_SIZE_WAIT_CALL || !rcu_cpu_beenfullyonline(raw_smp_processor_id()))
 		sdp = per_cpu_ptr(ssp->sda, get_boot_cpu_id());
 	else
 		sdp = raw_cpu_ptr(ssp->sda);