Re: the stuttering regression in 7.0: should I have done something different?

[Tony Rodriguez]

Hi Thomas,

Thank you for the detailed analysis — this helps clarify the situation 
on the SPARC side.

You are correct that my earlier explanation focused too much on the core 
changes and not enough on the SPARC clockevents behaviour under the new 
forced-min-delta semantics. However, having a stable system is equally 
important and is the main reason that I developed the test patch (to 
help). However, your explanation makes sense.

I will apply your debug patch and capture the trace as requested.  If 
the system becomes unresponsive, I will enable ftrace_dump_on_oops so 
the trace is emitted when the hung task detector triggers.

Once I have the trace, I’ll send another email.

Thanks again for the guidance — I’ll follow up with trace results 
sometime tomorrow.

Tony

On 5/10/26 2:29 PM, Thomas Gleixner wrote:
> On Fri, May 08 2026 at 13:15, Tony Rodriguez wrote:
> > Just confirmed on my end today.  This regression also impacts both
> > SPARC64 S7-2 and SPARC64 T7-1 on v7.0.4 and v7.1-rc2 as well. Different
> > systems using the same exact kernels.
> >
> > ** Please see points (A1) (A2) (B1) (B2)
> >
> > Once again, I am not experiencing such issues when "my patch" (link
> > below) is added to address this regression.
> >
> > https://github.com/sparclinux/issues/issues/79#issuecomment-4362173884
> Github issues are really not helpful.
>
> > PS - On May 2nd 2026 at 9:42 PM: I also sent an email to Thomas Gleixner
> > regarding this issue.  Will be happy to validate any patches from your
> > end regarding this issue, as time permits me to do so.
> Sorry, that mail got lost as it was in reply to a random other archived
> thread which has absolutely nothing to do with the problem at hand.
>
> I just looked at your github thing. Despite your changelog claiming
> otherwise your "fix" breaks the DoS protection completely. It's a
> polished version of a revert.
>
> It also lacks a proper root cause analysis. This list:
>
>      - skipped programming events when delta <= min_delta_ns
>      - changed force semantics for overdue events
>      - introduced a sticky next_event_forced state
>      - returned success even when no event was programmed
>
> does not qualify and is actually wrong.
>
> The code does not unconditionally skip the programming of events when
> delta <= min_delta_ns. It only does so conditionally when the previous
> force programmed min_delta_ns event has not been delivered to the kernel
> yet, i.e. dev->next_event_forced is still set.
>
> That flag is only set when the minimal value has been successfully
> programmed and it _is_ cleared on the next timer interrupt, which should
> obviously happen due to this minimal delta programming. It is also
> cleared when a new event > min_delta_ns is successfully programmed
> _before_ the previous one was delivered.
>
> IOW, the core code programmed the hardware with the min_delta_ns
> (min_delta_ticks) timeout and the SPARC clockevents driver returned
> success (0). Now the core code refuses to do further reprogramming with
> the min_delta_ns timeout as that would shift the expiry (interrupt)
> further out until the interrupt actually is delivered or some other
> event which is not below the min_delta_ns threshold is programmed.
>
> So let's assume that this logic is causing the problem, then the only
> explanation for the observed behaviour is that the expected interrupt
> due to a forced min_delta_ns programming is never delivered.
>
> That made me look into the SPARC specific set_next_event() functions. I
> don't know which variant your machines are using, but all of them have
> the same underlying problem. The interrupt is based on a equal
> comparator, so the programming logic for each of the tick variants is:
>
> $variant_add_compare(delta)
> {
>        cmp = read_timer() + delta;
>        write_comparator(cmp);
>        now = read_timer();
>        return (now - cmp) > 0;
> }
>
> and the actual set_next_event() function which is invoked from the core
> code does:
>
>     return tick_operations.add_compare(delta) ? -ETIME : 0;
>
> IOW, when the timer read _after_ writing the comparator value is ahead
> of the comparator value the operation failed. Looks about right in
> theory.
>
> But then there is the reality of hardware which ruins everything. I've
> banged my head against the wall many years ago when debugging a similar
> issue with the x86 HPET which has the same hardware design failure of
> using a compare equal comparator instead of having a compare less than
> equal one. See the lengthy comment in hpet_clkevt_set_next_event() for
> further information.
>
> Can you apply the debug patch below, which will disable tracing once it
> hits the hung task detector and then retrieve the trace?
>
> If that's not possible as the system is unresponsive, then please add
> 'ftrace_dump_on_oops' on the kernel command line or enable it after boot
> in /proc/sys/kernel and let the kernel panic when it hits the hung task
> detector fail.
>
> Thanks,
>
>          tglx
> ---
> --- a/arch/sparc/kernel/time_64.c
> +++ b/arch/sparc/kernel/time_64.c
> @@ -732,8 +732,10 @@ void __irq_entry timer_interrupt(int irq
>   	if (unlikely(!evt->event_handler)) {
>   		printk(KERN_WARNING
>   		       "Spurious SPARC64 timer interrupt on cpu %d\n", cpu);
> -	} else
> +	} else {
> +		trace_printk("Invoking handler %pS\n", evt->event_handler);
>   		evt->event_handler(evt);
> +	}
>   
>   	irq_exit();
>   
> --- a/kernel/hung_task.c
> +++ b/kernel/hung_task.c
> @@ -248,6 +248,7 @@ static void hung_task_info(struct task_s
>   	 * accordingly
>   	 */
>   	if (sysctl_hung_task_warnings || hung_task_call_panic) {
> +		tracing_off();
>   		if (sysctl_hung_task_warnings > 0)
>   			sysctl_hung_task_warnings--;
>   		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
> --- a/kernel/time/clockevents.c
> +++ b/kernel/time/clockevents.c
> @@ -370,18 +370,22 @@ int clockevents_program_event(struct clo
>   		delta = min(delta, (int64_t) dev->max_delta_ns);
>   		cycles = ((u64)delta * dev->mult) >> dev->shift;
>   		if (!dev->set_next_event((unsigned long) cycles, dev)) {
> +			trace_printk("Successfully programmed %lld %lld\n", expires, delta);
>   			dev->next_event_forced = 0;
>   			return 0;
>   		}
>   	}
>   
> -	if (dev->next_event_forced)
> +	if (dev->next_event_forced) {
> +		trace_printk("Skipping %lld %lld\n", expires, delta);
>   		return 0;
> +	}
>   
>   	if (dev->set_next_event(dev->min_delta_ticks, dev)) {
>   		if (!force || clockevents_program_min_delta(dev))
>   			return -ETIME;
>   	}
> +	trace_printk("Force programmed min delta %lld %lld\n", expires, delta);
>   	dev->next_event_forced = 1;
>   	return 0;
>   }