[PATCH 1/6] sched/isolation: Consider nohz_full CPUs isolated in cpu_is_isolated() again

[Matthew Ruffell]

From: Caron Wills <caron.wills@xxxxxxxxxx>

Commit 1c3329df409b ("sched/isolation: Remove HK_TYPE_TICK test from
cpu_is_isolated()") reduced cpu_is_isolated() to a HK_TYPE_DOMAIN test on
the assumption that "it doesn't make sense to use nohz_full without also
isolating the related CPUs from the domain topology", i.e. that every
nohz_full CPU is also domain isolated.

The kernel does not enforce that assumption. "nohz_full=<list>" and
"isolcpus=nohz,<list>" are accepted and documented on their own, and
deployments exist that deliberately keep the nohz_full CPUs inside the
scheduler domains, for example the AWS c8a.metal-48xl systems of
LP: #2165410, booted with

  isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178 rcu_nocbs=18-95,108-178

On such a system cpu_is_isolated() is now false for every CPU and all of
its users silently lost their isolation awareness compared to v6.8, which
also tested HK_TYPE_TICK:

  - fs/buffer.c bh_lru_install() caches buffer_heads on the nohz_full
    CPUs again, so has_bh_in_lru() makes __lru_add_drain_all() queue
    drain work on them;
  - mm/vmstat.c vmstat_shepherd() queues vmstat_update on them again;
  - mm/memcontrol.c schedule_drain_work() queues memcg stock drains on
    them again;
  - block/blk-mq.c keeps them in hctx->cpumask.

Per-CPU work queued on a nohz_full CPU that runs a SCHED_FIFO busy loop
never executes, and the flush_work() callers hang behind it. This is one
half of the PID 1 D-state stall described in LP: #2165410.

Test HK_TYPE_KERNEL_NOISE (the housekeeping type set by nohz_full= and
isolcpus=nohz, of which HK_TYPE_TICK is an alias) in addition to
HK_TYPE_DOMAIN, and use the housekeeping_cpu() wrapper so that systems
without any CPU isolation keep paying nothing but a patched-out static
branch. On systems where the nohz_full CPUs are also domain isolated
nothing changes.

Fixes: 1c3329df409b ("sched/isolation: Remove HK_TYPE_TICK test from cpu_is_isolated()")
Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@xxxxxxxxxx>
Signed-off-by: Matthew Ruffell <matthew.ruffell@xxxxxxxxxxxxx>
---
 include/linux/sched/isolation.h | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h
index cf0fd03dd7a2..294c508b0018 100644
--- a/include/linux/sched/isolation.h
+++ b/include/linux/sched/isolation.h
@@ -84,9 +84,17 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type)
 	return true;
 }
 
+/*
+ * A CPU is isolated if it has been taken out of the scheduler domains
+ * (isolcpus=domain or a cpuset isolated partition) or out of kernel noise
+ * (nohz_full= or isolcpus=nohz). The latter does not imply the former:
+ * the kernel accepts nohz_full CPUs that are still part of the scheduler
+ * domains, and deferred per-CPU work must not target those CPUs either.
+ */
 static inline bool cpu_is_isolated(int cpu)
 {
-	return !housekeeping_test_cpu(cpu, HK_TYPE_DOMAIN);
+	return !housekeeping_cpu(cpu, HK_TYPE_DOMAIN) ||
+	       !housekeeping_cpu(cpu, HK_TYPE_KERNEL_NOISE);
 }
 
 #endif /* _LINUX_SCHED_ISOLATION_H */
-- 
2.55.0