Re: [PATCH v8] lib/group_cpus: Snapshot cluster masks to keep grouping hotplug invariant

[Andrew Morton]

On Sat, 14 Nov 2026 20:44:03 +0530 Vishal Badole <Vishal.Badole@xxxxxxx> wrote:

> group_cpus_evenly() builds the managed-IRQ affinity spread used by
> multi-queue devices such as NVMe. That spread is meant to be a property
> of the static CPU topology: it walks cpu_present_mask and then
> cpu_possible_mask so every hardware queue owns a fixed set of CPUs,
> including CPUs that are offline at the time. A driver depends on that
> partition staying stable across re-computation - the CPUs a queue is
> given at probe must still describe the same queue after the device is
> later reset and its affinity recomputed.
> 
> On an AMD system that stability breaks across an s2idle cycle. CPUs 3-11
> are offlined before suspend, leaving only CPUs 0-2 online. s2idle itself
> does not offline secondary CPUs. The NVMe controller uses the
> simple-suspend quirk, so resume fully re-initialises it and recomputes the
> affinity spread while CPUs 3-11 remain offline. The system then hangs for
> roughly two minutes and stays sluggish afterwards, the controller only
> making progress through its command-timeout poll:
> 
>   nvme nvme0: I/O tag 898 (3382) QID 9 timeout, completion polled
>   nvme nvme0: I/O tag 398 (618e) QID 11 timeout, completion polled
> 
> QID 9 and QID 11 are the queues whose CPUs were offline when the spread
> was recomputed. "completion polled" means the commands did finish in
> hardware, but their interrupts were never delivered to a CPU that was
> watching the queue, so nothing reaped them until the timeout fired.
> 
> It happens because commit 89802ca36c96 ("lib/group_cpus: make group CPU
> cluster aware") derives the cluster groups from topology_cluster_cpumask(),
> which lists only the cluster siblings that are online when it is called.
> The resulting partition therefore depends on the transient online mask
> rather than on the topology alone. Recomputed on resume while CPUs 3-11
> remain offline, it no longer matches the boot-time partition, and a
> queue is left with an affinity that does not cover the CPU it is meant to
> serve once that CPU comes back online. The dependence is on the online
> mask, not on any AMD-specific behaviour, so the same stall is reproducible
> on Intel platforms as well.
> 
> Fix this by snapshotting cluster and SMT masks together under
> cpus_read_trylock() when all present CPUs are online. Latch the first
> capture attempt, permanently using CPU-order grouping if it fails, and
> propagate later allocation errors instead of switching algorithms. This
> keeps successful spreads stable across CPU offline/online cycles for an
> unchanged group count, present/possible CPU sets and NUMA topology.

updated, thanks.

> Changes in v8:
> - Check for a snapshot before allocating the temporary cluster mask.

here's the v7->v8 delta:

 lib/group_cpus.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

--- a/lib/group_cpus.c~lib-group_cpus-snapshot-cluster-masks-to-keep-grouping-hotplug-invariant-v8
+++ a/lib/group_cpus.c
@@ -380,9 +380,6 @@ static int alloc_cluster_groups(unsigned
 	const struct cpumask **clusters;
 	struct node_groups *cluster_groups;
 
-	if (!cluster_snapshot)
-		goto no_cluster;
-
 	cpumask_copy(msk, node_cpumask);
 
 	/* Probe how many clusters in this node. */
@@ -454,6 +451,9 @@ static int __try_group_cluster_cpus(unsi
 	cpumask_var_t nmsk;
 	unsigned int i, nc;
 
+	if (!cluster_snapshot)
+		return 0;
+
 	if (!zalloc_cpumask_var(&nmsk, GFP_KERNEL))
 		return -ENOMEM;
 
_