Re: [REGRESSION] sched/rt: NO_RT_PUSH_IPI causes multi-second PI-boost starvation in pro-audio workloads (dd29c017aed6)

[Thorsten Leemhuis]

On 8/15/26 00:36, Steven Rostedt wrote:
> On Fri, 14 Aug 2026 16:12:35 -0500
> Martin King <martin3kx@xxxxxxxxx> wrote:
> 
>> Commit dd29c017aed6 ("sched/rt: Have RT_PUSH_IPI be default off for non
>> PREEMPT_RT") causes reproducible multi-second audio dropouts on an ordinary
>> non-PREEMPT_RT desktop running a DAW.
> 
> Thanks for the report.

This made it to my list of tracked regression -- but it looks like there
was no progress for a while. Is that the case? Steven, did that maybe
fall through the cracks due to the travel you mentioned? Or was this
resolved meanwhile and I just missed it?

Ciao, Thorsten

>> Symptom
>> -------
>> Audio cuts to silence for 0.2-3.5s at irregular intervals (roughly one per
>> 2-3 minutes of use) while the machine is otherwise idle and responsive.
>>
>> Mechanism (as measured, not inferred)
>> -------------------------------------
>> REAPER runs one SCHED_RR prio 20 audio thread plus ~15 SCHED_OTHER "livefx"
>> worker threads. During a stall:
>>
>>   - the RT audio thread is blocked in rt_mutex_schedule() on a PI mutex
>>   - exactly ONE livefx worker holds that mutex, has been PI-boosted to RT,
>>     and is enqueued but never scheduled
>>   - its schedstat shows run_ns ~0 with wait_ns tracking the entire stall:
>>
>>         reaper/livefx tid=43910  run=0.0ms  wait=3406.6ms  slices=2
>>         (during a 3403ms dropout)
>>
>>   - meanwhile ~17 sibling threads run normally with wait_ns = 0, and no
>>     thread does meaningful work (largest run was 7.2ms across 3.4s)
>>   - the machine stays responsive throughout; unrelated SCHED_OTHER work
>>     proceeds at 84-94% of normal rate with 0-1ms runqueue wait
>>   - all 16 cores keep taking timer interrupts at normal rates
>>
>> So a single PI-boosted RT task sits on a runqueue, unmigrated, while other
>> cores idle. With push disabled nothing hands it to an idle CPU.
>>
>> Evidence
>> --------------------------------
>>   kernel  RT_PUSH_IPI  playing   dropouts >=25ms          xruns
>>   7.1.8   off (default)  19 min   8  (191ms - 3471ms)   bursts
>> 23/265/434 per min
>>   7.1.3   on             39 min   0                     0
>>   7.1.8   on             42 min   0                     0
>>
>> The third row differs from the first ONLY in the sched feature: Identical
>> kernel, toolchain, GPU driver, project and audio configuration. At the
>> observed broken rate ~13 dropouts were expected in that window; zero were
>> seen, at both 25ms and 10ms detection thresholds (10ms is ~2 missed audio
>> cycles at quantum 256 / 48kHz).
>>
>> Workaround
>> ----------
>>   echo RT_PUSH_IPI > /sys/kernel/debug/sched/features
>>
>> Applied at boot via a systemd unit.
>>
>> System
>> ------
>>   AMD Ryzen 7 5800X (8c/16t), Arch Linux
>>   kernel 7.1.8-arch1-3 (and 7.1.5, 7.1.6 - all affected; 7.1.3 is clean)
>>   PipeWire 1.6.8 + pipewire-jack, Focusrite Scarlett 18i20, quantum 256 @ 48kHz
>>   CONFIG_PREEMPT_RT not set
> 
> Unfortunately, I'm going to be traveling next week and I may not be able to
> look into this until I get back. I don't see anything that the normal (non
> PUSH) logic would fail in this case. Thus I'm thinking it simply has a bug
> that has been hiding because we've been using the PUSH logic.
> 
> I may be asking for some traces here when I get back to find out the exact
> cause. Use the workaround for now.
> 
> The RT_PUSH_IPI was added because of a "thundering herd" of IPIs when all
> but one CPU dropped its priority and the one still running had more than
> one RT task on it. That would cause all the other CPUs to send an IPI to
> that CPU to do a pull, but only would would get it. The RT_PUSH_IPI was
> created to pass around a single IPI to do the push for all CPUs wanting a
> new RT task to run.
> 
> The above use case doesn't look like it is having that. Thus, I'm thinking
> something else broke.
> 
> -- Steve
>