Re: [REGRESSION] ACPI: threaded SCI handler causes soft lockup in kdump capture kernel

[Mario Limonciello]

On 6/2/26 12:17, Rafael J. Wysocki wrote:
> On Monday, June 1, 2026 10:29:08 PM CEST Mohan Yelugoti wrote:
> > Hi,
> >
> > We ship SONiC[0] on top of Debian stable with a small set of patches
> > on top (see [1]). After moving SONiC from Debian bookworm
> > to trixie, we started seeing a regression where the kdump capture
> > kernel hits a soft lockup while collecting the crash dump, so no
> > vmcore is produced.
> >
> > We trigger the original panic in the kernel with:
> >
> >        echo c > /proc/sysrq-trigger
> >
> > The capture kernel then boots and hangs. To capture a usable trace
> > rather than waiting indefinitely, I added the following to the
> > capture-kernel command line so that it panics on soft lockup (and
> > dumps backtraces from all CPUs) when the hang is detected:
> >
> >        debug=1 loglevel=7 softlockup_all_cpu_backtrace=1 softlockup_panic=1
> >
> > With those in place, the capture kernel prints:
> >
> >        watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [irq/9-acpi:39]
> >        CPU: 0 UID: 0 PID: 39 Comm: irq/9-acpi Not tainted
> >          6.12.41+deb13-sonic-amd64 #1 Debian 6.12.41-1
> >        RIP: 0010:acpi_os_read_port+0x30/0xa0
>
> I have no idea what possibly may lock up in that function, sorry.
>
> In this particular case it is reading a byte from the I/O space via inb(),
> that's all.
>
> I'm wondering if it gets interrupted for some reason and then something
> odd happens that prevents it from continuing.
>
> >        Call Trace:
> >         <TASK>
> >         acpi_hw_gpe_read+0x61/0x80
> >         acpi_ev_detect_gpe+0x74/0x180
> >         acpi_ev_gpe_detect+0xe1/0x130
> >         acpi_ev_sci_xrupt_handler+0x1d/0x40
> >         acpi_irq+0x1c/0x40
> >         irq_thread_fn+0x23/0x60
> >         irq_thread+0x1b3/0x2f0
> >         kthread+0xd2/0x100
> >         ret_from_fork+0x34/0x50
> >         ret_from_fork_asm+0x1a/0x30
> >         </TASK>
> >        Kernel panic - not syncing: softlockup: hung tasks
> >
> > Between the bookworm and trixie kernels, the ACPI SCI handler was
> > moved from a hardirq handler to a threaded handler by:
> >
> >      7a36b901a6eb ("ACPI: OSL: Use a threaded interrupt handler for SCI")
>
> Maybe it exposed some latent issue.
>
> It's too late to revert it from the mainline anyway.
>
> > To confirm this is the trigger, I reverted that commit on top of
> > 6.12.41 and re-ran the same sysrq-trigger reproducer. The capture
> > kernel then completes the crash dump without tripping the soft
> > lockup watchdog.
> >
> > Happy to test patches or gather more data as this is reproducible
> > consistently on our systems.
>
> Does it always occur at the same spot?  If so, you could try to
> instrument the code around it and look for clues.

Especially if it's just a latent exposed issue before spending too much 
time intrumenting it's probably worth cross referencing 7.1-rc6 as well 
to see if the same issue can occur.

> > This issue has been raised in the SONiC project at [2].
> >
> >    Context
> >    -------
> > - Kernel:        6.12.41+deb13-sonic-amd64 (Debian 6.12.41-1);
> > - Last known good: Debian bookworm kernel (6.1.123), predates
> >                       7a36b901a6eb.
> >
> > - Default capture-kernel cmdline (from /etc/default/kdump-tools,
> >    KDUMP_CMDLINE_APPEND):
> >
> >        irqpoll nr_cpus=1 nousb systemd.unit=kdump-tools.service
> >        ata_piix.prefer_ms_hyperv=0 panic=10 debug loglevel=4
> >        hpet=disable pcie_port=compat pci=nommconf
> >
> > For the trace above, loglevel was raised from 4 to 7 and the
> > following were added:
> >
> >       debug=1 softlockup_all_cpu_backtrace=1 softlockup_panic=1
> >
> > [0]: https://github.com/sonic-net
> > [1]: https://github.com/sonic-net/sonic-linux-kernel/tree/master/patches-sonic
> > [2]: https://github.com/sonic-net/sonic-linux-kernel/pull/580