Re: [PATCH] clocksource: Use the clocksource under test for the skew check timeout

[Thomas Gleixner]

On Mon, Nov 02 2026 at 16:45, Mani Tofigh via wrote:
> watchdog_check_skew() waits for the remote CPU with interrupts disabled
> and gives up after a timeout measured with ktime_get_raw_fast_ns().
> While the clocksource is still jiffies during boot (or with
> clocksource=jiffies) that timeout can't expire, because only the boot
> CPU's tick advances jiffies and that's the CPU spinning with interrupts
> disabled.
>
> The boot then hangs when the remote CPU can't answer, eg in the
> stop_machine() of the switch away from jiffies:
>
>   CPU0                                  CPU1
>   multi_cpu_stop()                      multi_cpu_stop()
>                                           local_irq_disable()
>                                           /* waits for CPU0 */
>     <timer interrupt>
>       clocksource_watchdog()
>         /* sends CSD to CPU1 */
>         watchdog_check_skew()
>           /* waits for CPU1 forever */
>
> Nothing gets printed.  With the hard lockup detector on in a KVM guest:

The above explanation is enough. No need for this backtrace.

> Measure the timeout with the clocksource under test instead, which
> keeps counting with interrupts disabled.  With the fix, the watchdog
> gives up after the timeout and the boot goes on.  A 2-vCPU guest on one
> host core hung in 4 of 150 boots without it, and it didn't hang at all
> in 400 tests with the fix applied.
>
> This bug was identified and confirmed with the help of a deterministic
> hypervisor currently under development.  The C reproducer below however
> hits it within about 80s on a kernel booted with clocksource=jiffies.
>
> Fixes: 763aacf86f1b ("clocksource: Rewrite watchdog code completely")
> Cc: stable@xxxxxxxxxxxxxxx
> Signed-off-by: Mani Tofigh <mt4022@xxxxxxxxxxxx>
> ---
> Srikanth reported silent early boot hangs of large KVM guests that he
> bisected to the same commit [1].  I think it's the same bug but it was
> never confirmed.  Srikanth, would you mind giving this a try on your
> setup?

Yes. I gave him a similar patch which simply uses sched_clock()
instead. That solves the problem, but I was reluctant to post it due to
the lack of explanation why this ends up in a situation which does not
allow to make progress.

So your reproducer does:

  while true; do echo 0 >/sys/devices/system/cpu/cpu1/online; echo 1 >/sys/devices/system/cpu/cpu1/online; done

right?

Unfortunately it does not reproduce here within a reasonable amount of
time (still going strong after one hour...).

But yes, your explanation makes sense and triggering it with the hotplug
code is obviously a matter of probability.

Though one thing I thought about before looking at your mail while
reasoning over the possible causes of Srikanth's report and the possible
consequences is that such a timeout might in the worst case result in a
stale CPU state which is silently ignored. The consequence is that the
watchdog wont run on a CPU with such state ever again.

So something like the below delta patch on top of your fix (or the
sched_clock()) variant might be appropriate.

Can you and Srikanth please run that on top to validate that we don't
end up with that?

Thanks,

        tglx
---
--- a/kernel/time/clocksource.c
+++ b/kernel/time/clocksource.c
@@ -250,6 +250,7 @@ enum wd_result {
 	WD_FREQ_RESET,
 	WD_FREQ_SKEWED,
 	WD_CPU_TIMEOUT,
+	WD_CPU_STALLED,
 	WD_CPU_SKEWED,
 };
 
@@ -398,7 +399,7 @@ static void __watchdog_check_cpu_skew(st
 
 	wd = per_cpu_ptr(&watchdog_cpu_data, cpu);
 	if (atomic_read(&wd->remote_inprogress) || wd_csd_locked(wd)) {
-		watchdog_data.result = WD_CPU_TIMEOUT;
+		watchdog_data.result = WD_CPU_STALLED;
 		return;
 	}
 
@@ -590,11 +591,16 @@ static void watchdog_print_freq_skew(str
 	pr_warn("Clocksource %20s interval: %16lluns\n", cs->name, watchdog_data.cs_delta);
 }
 
-static void watchdog_handle_remote_timeout(struct clocksource *cs)
+static void watchdog_print_remote_timeout(struct clocksource *cs)
 {
 	pr_info_once("Watchdog remote CPU %u read timed out\n", watchdog_data.curr_cpu);
 }
 
+static void watchdog_print_remote_stall(struct clocksource *cs)
+{
+	pr_warn("Watchdog remote CPU %u stalled\n", watchdog_data.curr_cpu);
+}
+
 static void watchdog_print_remote_skew(struct clocksource *cs)
 {
 	pr_warn("Marking clocksource %s unstable due to inter CPU skew\n", cs->name);
@@ -635,9 +641,13 @@ static void watchdog_check_result(struct
 
 	case WD_CPU_TIMEOUT:
 		/* Remote check timed out. Try again next cycle. */
-		watchdog_handle_remote_timeout(cs);
+		watchdog_print_remote_timeout(cs);
 		return;
 
+	case WD_CPU_STALLED:
+		watchdog_print_remote_stall(cs);
+		break;
+
 	case WD_CPU_SKEWED:
 		watchdog_print_remote_skew(cs);
 		break;