Re: [PATCH] clocksource: Use the clocksource under test for the skew check timeout

[Aithal, Srikanth]

On 11/4/2026 1:50 AM, Thomas Gleixner wrote:
> On Mon, Nov 02 2026 at 16:45, Mani Tofigh via wrote:
> > watchdog_check_skew() waits for the remote CPU with interrupts disabled
> > and gives up after a timeout measured with ktime_get_raw_fast_ns().
> > While the clocksource is still jiffies during boot (or with
> > clocksource=jiffies) that timeout can't expire, because only the boot
> > CPU's tick advances jiffies and that's the CPU spinning with interrupts
> > disabled.
> >
> > The boot then hangs when the remote CPU can't answer, eg in the
> > stop_machine() of the switch away from jiffies:
> >
> >    CPU0                                  CPU1
> >    multi_cpu_stop()                      multi_cpu_stop()
> >                                            local_irq_disable()
> >                                            /* waits for CPU0 */
> >      <timer interrupt>
> >        clocksource_watchdog()
> >          /* sends CSD to CPU1 */
> >          watchdog_check_skew()
> >            /* waits for CPU1 forever */
> >
> > Nothing gets printed.  With the hard lockup detector on in a KVM guest:
> The above explanation is enough. No need for this backtrace.
>
> > Measure the timeout with the clocksource under test instead, which
> > keeps counting with interrupts disabled.  With the fix, the watchdog
> > gives up after the timeout and the boot goes on.  A 2-vCPU guest on one
> > host core hung in 4 of 150 boots without it, and it didn't hang at all
> > in 400 tests with the fix applied.
> >
> > This bug was identified and confirmed with the help of a deterministic
> > hypervisor currently under development.  The C reproducer below however
> > hits it within about 80s on a kernel booted with clocksource=jiffies.
> >
> > Fixes: 763aacf86f1b ("clocksource: Rewrite watchdog code completely")
> > Cc: stable@xxxxxxxxxxxxxxx
> > Signed-off-by: Mani Tofigh <mt4022@xxxxxxxxxxxx>
> > ---
> > Srikanth reported silent early boot hangs of large KVM guests that he
> > bisected to the same commit [1].  I think it's the same bug but it was
> > never confirmed.  Srikanth, would you mind giving this a try on your
> > setup?
> Yes. I gave him a similar patch which simply uses sched_clock()
> instead. That solves the problem, but I was reluctant to post it due to
> the lack of explanation why this ends up in a situation which does not
> allow to make progress.
>
> So your reproducer does:
>
>    while true; do echo 0 >/sys/devices/system/cpu/cpu1/online; echo 1 >/sys/devices/system/cpu/cpu1/online; done
>
> right?
>
> Unfortunately it does not reproduce here within a reasonable amount of
> time (still going strong after one hour...).
>
> But yes, your explanation makes sense and triggering it with the hotplug
> code is obviously a matter of probability.
>
> Though one thing I thought about before looking at your mail while
> reasoning over the possible causes of Srikanth's report and the possible
> consequences is that such a timeout might in the worst case result in a
> stale CPU state which is silently ignored. The consequence is that the
> watchdog wont run on a CPU with such state ever again.
>
> So something like the below delta patch on top of your fix (or the
> sched_clock()) variant might be appropriate.
>
> Can you and Srikanth please run that on top to validate that we don't
> end up with that?
>
> Thanks,
>
>          tglx
> ---
> --- a/kernel/time/clocksource.c
> +++ b/kernel/time/clocksource.c
> @@ -250,6 +250,7 @@ enum wd_result {
>   	WD_FREQ_RESET,
>   	WD_FREQ_SKEWED,
>   	WD_CPU_TIMEOUT,
> +	WD_CPU_STALLED,
>   	WD_CPU_SKEWED,
>   };
>   
> @@ -398,7 +399,7 @@ static void __watchdog_check_cpu_skew(st
>   
>   	wd = per_cpu_ptr(&watchdog_cpu_data, cpu);
>   	if (atomic_read(&wd->remote_inprogress) || wd_csd_locked(wd)) {
> -		watchdog_data.result = WD_CPU_TIMEOUT;
> +		watchdog_data.result = WD_CPU_STALLED;
>   		return;
>   	}
>   
> @@ -590,11 +591,16 @@ static void watchdog_print_freq_skew(str
>   	pr_warn("Clocksource %20s interval: %16lluns\n", cs->name, watchdog_data.cs_delta);
>   }
>   
> -static void watchdog_handle_remote_timeout(struct clocksource *cs)
> +static void watchdog_print_remote_timeout(struct clocksource *cs)
>   {
>   	pr_info_once("Watchdog remote CPU %u read timed out\n", watchdog_data.curr_cpu);
>   }
>   
> +static void watchdog_print_remote_stall(struct clocksource *cs)
> +{
> +	pr_warn("Watchdog remote CPU %u stalled\n", watchdog_data.curr_cpu);
> +}
> +
>   static void watchdog_print_remote_skew(struct clocksource *cs)
>   {
>   	pr_warn("Marking clocksource %s unstable due to inter CPU skew\n", cs->name);
> @@ -635,9 +641,13 @@ static void watchdog_check_result(struct
>   
>   	case WD_CPU_TIMEOUT:
>   		/* Remote check timed out. Try again next cycle. */
> -		watchdog_handle_remote_timeout(cs);
> +		watchdog_print_remote_timeout(cs);
>   		return;
>   
> +	case WD_CPU_STALLED:
> +		watchdog_print_remote_stall(cs);
> +		break;
> +
>   	case WD_CPU_SKEWED:
>   		watchdog_print_remote_skew(cs);
>   		break;


Tested on 6628e7d4c6b25 / v7.4-rc1:

1. Mani's skew-timeout change + your WD_CPU_STALLED delta
2. Mani's patch alone (without the STALLED delta)
Both passed on my large-KVM recreation setup (5× smp=255 SVM guests, 
~2500 reboot stress).

Without these patches, the issue is still recreatable on 6628e7d4c6b25 / 
v7.4-rc1.