[PATCH 2/6] kthread: Keep unbound kthreads off nohz_full CPUs

[Matthew Ruffell]

From: Caron Wills <caron.wills@xxxxxxxxxx>

Since commit 041ee6f3727a ("kthread: Rely on HK_TYPE_DOMAIN for
preferred affinity management") kthread_fetch_affinity() only removes
domain isolated CPUs from a kthread's affinity, assuming that nohz_full
CPUs are always domain isolated as well. Up to v6.13 every kthread
inherited kthreadd's affinity to housekeeping_cpumask(HK_TYPE_KTHREAD),
and from v6.14 the preferred affinity was filtered with HK_TYPE_KTHREAD,
so nohz_full CPUs were always excluded.

On a system that keeps its nohz_full CPUs in the scheduler domains, e.g.

  isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178

HK_TYPE_DOMAIN covers every CPU, and the RCU expedited grace period
workers rcu_exp_par_gp_kthread_worker/N, the RCU boost kthreads rcub/N
and every NUMA node affine kthread can now land on nohz_full CPUs.
rcu_thread_affine_rnp() prefers the CPUs of the rcu_node, so for a leaf
rcu_node made entirely of nohz_full CPUs the worker is pinned to nothing
but nohz_full CPUs.

Those workers are SCHED_NORMAL (CONFIG_RCU_BOOST=n leaves kthread_prio
at 0), so a SCHED_FIFO userspace busy loop on such a CPU starves them
indefinitely. Every synchronize_rcu_expedited() in the system then
blocks in sync_rcu_exp_select_cpus() -> kthread_flush_work() until the
FIFO task yields. LP: #2165410 captured exactly that on three 7.0
kernels: the cpuset_migrate_mm kworker sits in

  synchronize_rcu_expedited+0x1b2/0x220
  lru_cache_disable+0x15/0x30
  do_migrate_pages+0x4e/0x2b0
  cpuset_migrate_mm_workfn+0x26/0x50

with rcu_exp_gp_kthread_worker in D state in kthread_flush_work(), and
PID 1 waits behind that migration in flush_workqueue() for minutes.

Intersect the affinity with the HK_TYPE_KERNEL_NOISE housekeeping mask
as well and fall back to the CPUs that are housekeeping for both types.
On systems where nohz_full implies domain isolation the two masks are
identical and nothing changes.

HK_TYPE_KERNEL_NOISE is named explicitly rather than HK_TYPE_KTHREAD:
commit 8f78b749f3da ("sched/isolation: Make HK_TYPE_KTHREAD an alias of
HK_TYPE_DOMAIN") (v7.1-rc3) turned the latter into a synonym of the domain
mask, which is precisely the assumption being corrected here, and made a
first version of this patch a no-op on 7.1+ kernels.

Fixes: 041ee6f3727a ("kthread: Rely on HK_TYPE_DOMAIN for preferred affinity management")
Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@xxxxxxxxxx>
Signed-off-by: Matthew Ruffell <matthew.ruffell@xxxxxxxxxxxxx>
---
 kernel/kthread.c | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)

diff --git a/kernel/kthread.c b/kernel/kthread.c
index fdb8fb07f40b..b9d8dfe1b6ca 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -327,24 +327,46 @@ void __noreturn kthread_complete_and_exit(struct completion *comp, long code)
 }
 EXPORT_SYMBOL(kthread_complete_and_exit);
 
+/*
+ * Unbound kthreads must run neither on domain isolated CPUs (isolcpus=domain,
+ * cpuset isolated partitions) nor on CPUs isolated from kernel noise
+ * (nohz_full=, isolcpus=nohz). The two sets usually coincide, but the kernel
+ * accepts nohz_full CPUs that remain in the scheduler domains, so both
+ * housekeeping types have to be honoured.
+ */
 static void kthread_fetch_affinity(struct kthread *kthread, struct cpumask *cpumask)
 {
-	const struct cpumask *pref;
+	const struct cpumask *pref, *hk_domain, *hk_kthread;
 
 	guard(rcu)();
 
+	hk_domain = housekeeping_cpumask(HK_TYPE_DOMAIN);
+	/*
+	 * Deliberately not HK_TYPE_KTHREAD: since v7.1 that is an alias of
+	 * HK_TYPE_DOMAIN (commit 8f78b749f3da), which is the very assumption
+	 * being corrected here.
+	 */
+	hk_kthread = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE);
+
 	if (kthread->preferred_affinity) {
 		pref = kthread->preferred_affinity;
 	} else {
 		if (kthread->node == NUMA_NO_NODE)
-			pref = housekeeping_cpumask(HK_TYPE_DOMAIN);
+			pref = hk_domain;
 		else
 			pref = cpumask_of_node(kthread->node);
 	}
 
-	cpumask_and(cpumask, pref, housekeeping_cpumask(HK_TYPE_DOMAIN));
+	cpumask_and(cpumask, pref, hk_domain);
+	cpumask_and(cpumask, cpumask, hk_kthread);
+	if (cpumask_empty(cpumask))
+		cpumask_and(cpumask, hk_domain, hk_kthread);
+	/*
+	 * Both the boot parameters and cpuset guarantee that at least one
+	 * CPU stays housekeeping for every type, but don't rely on it.
+	 */
 	if (!cpumask_intersects(cpumask, cpu_online_mask))
-		cpumask_copy(cpumask, housekeeping_cpumask(HK_TYPE_DOMAIN));
+		cpumask_copy(cpumask, hk_domain);
 }
 
 static void kthread_affine_node(void)
-- 
2.55.0