[PATCH 4/6] sched/isolation: Drain per-CPU LRU batches when CPUs become isolated

[Matthew Ruffell]

From: Caron Wills <caron.wills@xxxxxxxxxx>

Isolated CPUs no longer batch folios in their per-CPU LRU caches, so
that __lru_add_drain_all() never has to queue work on them. Boot time
isolation (isolcpus=, nohz_full=) takes effect before any folio is
batched, but a cpuset isolated partition can be created at runtime and
the affected CPUs may still hold folios in their batches from before.
Those would linger until the CPU happens to drain locally, and the next
lru_add_drain_all() would fall back to queueing work on the now isolated
CPU, which is exactly what must be avoided.

Drain all batches once from housekeeping_update(), right after the
HK_TYPE_DOMAIN mask has been updated and next to the vmstat and memcg
workqueue flushes that exist for the same reason, so that newly isolated
CPUs start out with empty batches.

Link: https://bugs.launchpad.net/bugs/2165410
Signed-off-by: Caron Wills <caron.wills@xxxxxxxxxx>
Signed-off-by: Matthew Ruffell <matthew.ruffell@xxxxxxxxxxxxx>
---
 kernel/sched/isolation.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c
index 156025ef81b7..17d8171fe97b 100644
--- a/kernel/sched/isolation.c
+++ b/kernel/sched/isolation.c
@@ -10,6 +10,7 @@
 #include <linux/sched/isolation.h>
 #include <linux/llist.h>
 #include <linux/pci.h>
+#include <linux/swap.h>
 #include "sched.h"
 
 enum hk_flags {
@@ -149,6 +150,13 @@ int housekeeping_update(struct cpumask *isol_mask)
 	pci_probe_flush_workqueue();
 	mem_cgroup_flush_workqueue();
 	vmstat_flush_workqueue();
+	/*
+	 * Folios batched on the newly isolated CPUs before the update would
+	 * otherwise linger in their per-CPU LRU batches, since isolated CPUs
+	 * stop batching from now on (see lru_batching_bypassed()) and remote
+	 * draining is exactly what must be avoided on them.
+	 */
+	lru_add_drain_all();
 
 	err = workqueue_unbound_housekeeping_update(housekeeping_cpumask(HK_TYPE_DOMAIN));
 	WARN_ON_ONCE(err < 0);
-- 
2.55.0