[PATCH] timers/migration: Read jiffies only when the migrator walk needs it

[Breno Leitao]

run_local_timers() calls tmigr_requires_handle_remote() on every tick,
and the function starts with get_jiffies_update(). That reads
jiffies_seq and last_jiffies_update, which the timekeeping CPU rewrites
on every tick. As the ticks of all CPUs are aligned, every CPU takes
these cache misses while the timekeeper is writing the lines.

For an active CPU the value is only used by the migrator of its first
level group: tmigr_requires_handle_remote_up() stops the walk at once
on any other CPU. With eight CPUs per group, seven out of eight active
CPUs read jiffies and throw the value away.

Check the migrator first, as tmigr_handle_remote() already does, and
read jiffies only when the walk is going to use it.

Fleet profiles show get_jiffies_update() at up to 1.7% of non-idle
cycles on the largest AMD systems, almost all of it called from
tmigr_requires_handle_remote().

In a 72 vCPU KVM guest (AMD EPYC 9D85 host, HZ=1000) with every vCPU
spinning in user space, get_jiffies_update() calls drop from 367k to
46k per 5 seconds. tick_nohz_handler() duration drops also.

Signed-off-by: Breno Leitao <leitao@xxxxxxxxxx>
---
 kernel/time/timer_migration.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/kernel/time/timer_migration.c b/kernel/time/timer_migration.c
index f920e73fff5112..dfa936bc485f59 100644
--- a/kernel/time/timer_migration.c
+++ b/kernel/time/timer_migration.c
@@ -1195,7 +1195,6 @@ bool tmigr_requires_handle_remote(void)
 	if (tmigr_is_not_available(tmc))
 		return ret;
 
-	data.now = get_jiffies_update(&jif);
 	data.childmask = tmc->groupmask;
 	data.firstexp = KTIME_MAX;
 	data.check = false;
@@ -1208,11 +1207,21 @@ bool tmigr_requires_handle_remote(void)
 	 * set only by the local CPU.
 	 */
 	if (!tmc->idle) {
+		/*
+		 * Bail out early when not the migrator, to avoid reading
+		 * jiffies_seq, which the timekeeper writes on every tick.
+		 */
+		if (!tmigr_check_migrator(tmc->tmgroup, data.childmask))
+			return false;
+
+		data.now = get_jiffies_update(&jif);
 		__walk_groups(&tmigr_requires_handle_remote_up, &data, tmc);
 
 		return data.check;
 	}
 
+	data.now = get_jiffies_update(&jif);
+
 	/*
 	 * When the CPU is idle, compare @tmc->wakeup with @data.now. The lock
 	 * is required on 32bit architectures to read the variable consistently

---
base-commit: aaa2f954c21b205a0bcd0961e5b2be4e82e58455
base-tree: git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
change-id: 20261112-tmigr-lazy-jiffies-c5d2f4930095

Best regards,
--  
Breno Leitao <leitao@xxxxxxxxxx>