Re: [PATCH v3] sparc64: Fix comparator problem with timer interrupts

[Stian Halseth]

Hi,

I just got a SPARC T7-1 and have independent reproduction and
confirmation of the fix.

Without the patch the system is not really usable. I can just bring
down a network interface and that alone triggers a hang. 
Hung on 3 out of 3 boots unpatched:

  task:ip              state:D
   [<0000000000cfebc4>] wait_for_completion+0x64/0x180
   [<0000000000509c30>] synchronize_rcu_normal+0x150/0x280
   [<0000000010079060>] ixgbe_down+0x1a0/0x460 [ixgbe]
   [<000000001007a4c8>] ixgbe_close+0xe8/0x100 [ixgbe]
   [<0000000000af1cf8>] netif_change_flags+0x18/0x80
   [<0000000000b08618>] rtnl_newlink+0x758/0xaa0

A second caller hit the same stall under kernel build load, from
network namespace teardown rather than any driver path:

  task:kworker/u1024:0 state:D
   Workqueue: netns cleanup_net
   [<0000000000cfebc4>] wait_for_completion+0x64/0x180
   [<0000000000504ab4>] rcu_barrier+0x214/0x480
   [<0000000000af3f88>] netdev_run_todo+0x48/0x600
   [<0000000000ad99c8>] cleanup_net+0x1c8/0x360

Both are plain RCU waits:

  rcu: rcu_sched kthread timer wakeup didn't happen for 6002 jiffies!
       g193 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
  rcu:     Possible timer handling issue on cpu=20 timer-softirq=107
  rcu: rcu_sched kthread starved for 6004 jiffies! ->cpu=20


With the patch applied: 3 boots with a clean first ifdown on each, plus
6 consecutive interface down/up cycles within one boot, and no stall or
hung-task messages in any of them.

Tested-by: Stian Halseth <stian@xxxxxx>


On Mon, 2026-08-31 at 20:07 +0200, Stian Halseth wrote:
> From: Tony Rodriguez <unixpro1970@xxxxxxxxx>
> 
> The tick/stick/hbtick add_compare() implementations program the
> comparator and then check whether the write took effect in time:
> 
> 	exp = read_cnt() + delta_ticks;
> 	write_cmp(exp);
> 	return (read_cnt() - exp) > 0;
> 
> A nonzero return value means the expiry time was already reached
> before the comparator write could take effect, so the interrupt may
> never fire, and the caller retries with a new expiry:
> 
> 	return tick.add_compare(delta_ticks) ? -ETIME : 0;
> 
> The check only fails the write when the counter has advanced past the
> expiry time, but not when it is equal to it. In the equal case it is
> unknown whether the comparator write took effect before or after the
> counter reached the expiry value, so the compare match - and with it
> the timer interrupt - may have been missed. add_compare() then
> reports success, the caller does not retry, and the CPU is left with
> no pending timer interrupt.
> 
> This results in stalled hrtimers and RCU stalls / hangs under load,
> observed on SPARC S7-2 and T7-1 systems:
> 
>     rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
>     rcu: rcu_sched kthread timer wakeup didn't happen for 5259
> jiffies!
>     rcu:     Possible timer handling issue on cpu=100 timer-
> softirq=15
> 
> Treat counter == expiry as failure as well, so the caller retries
> with a new expiry time. After this change S7-2 and T7-1 systems no
> longer hang.
> 
> Diagnosed-by: Thomas Gleixner <tglx@xxxxxxxxxx>
> Link: https://lore.kernel.org/all/87tssb6olo.ffs@tglx/
> Link: https://lore.kernel.org/all/871pfcznw0.ffs@tglx/
> Link:
> https://lore.kernel.org/all/20260519022421.5978-1-unixpro1970@xxxxxxxxx/
> Signed-off-by: Tony Rodriguez <unixpro1970@xxxxxxxxx>
> [stian: rewrote the changelog per review of v2, retested]
> Signed-off-by: Stian Halseth <stian@xxxxxx>
> ---
> v3:
>  - rewrite the changelog: describe the write/readback ordering and
> the
>    direction of the equal-compare case per Thomas Gleixner's review
>    https://lore.kernel.org/all/878q9fxywc.ffs@tglx/
>    (the code change is identical to v2)
>  - add the Link: tags to the original debugging discussion
>  - picked up with Tony's agreement after v2 stalled
>   
> https://lore.kernel.org/all/10382506-3f57-4b33-8356-34f23fb5f153@xxxxxxxxx/
> 
> Tested on an UltraSPARC T4-1 (sun4v, stick variant): 9.1M hrtimer
> reprograms in 90s across 32 threads with 10-500us expiries, no
> stalls, no lost wakeups (worst oversleep 657us).
> 
>  arch/sparc/kernel/time_64.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/sparc/kernel/time_64.c
> b/arch/sparc/kernel/time_64.c
> --- a/arch/sparc/kernel/time_64.c
> +++ b/arch/sparc/kernel/time_64.c
> @@ -146,7 +146,7 @@
>  			     : "=r" (new_tick));
>  	new_tick &= ~TICKCMP_IRQ_BIT;
>  
> -	return ((long)(new_tick - (orig_tick+adj))) > 0L;
> +	return ((long)(new_tick - (orig_tick+adj))) >= 0L;
>  }
>  
>  static unsigned long tick_add_tick(unsigned long adj)
> @@ -277,7 +277,7 @@
>  			     : "=r" (new_tick));
>  	new_tick &= ~TICKCMP_IRQ_BIT;
>  
> -	return ((long)(new_tick - (orig_tick+adj))) > 0L;
> +	return ((long)(new_tick - (orig_tick+adj))) >= 0L;
>  }
>  
>  static unsigned long stick_get_frequency(void)
> @@ -411,7 +411,7 @@
>  
>  	val2 = __hbird_read_stick() & ~TICKCMP_IRQ_BIT;
>  
> -	return ((long)(val2 - val)) > 0L;
> +	return ((long)(val2 - val)) >= 0L;
>  }
>  
>  static unsigned long hbtick_get_frequency(void)
> --
> 2.53.0

Attachment: signature.asc
Description: This is a digitally signed message part