Re: [RFC PATCH] panic, printk, sys_info: Introduce crash_kexec_in_memory_sys_info

[Aaron Tomlin]

On Tue, Sep 22, 2026 at 05:29:45PM +0200, Petr Mladek wrote:
> Adding Guilherme and John into Cc.
> 
> On Wed 2026-09-16 16:15:46, Aaron Tomlin wrote:
> > When investigating kernel panics, capturing post-mortem diagnostic
> > telemetry (e.g. memory zone metrics, lock states, active timers, and
> > blocked tasks) is vital for root-cause analysis.
> >
> > While crash_kexec_post_notifiers allows executing panic notifiers and
> > sys_info() before jumping to the kdump kernel, it is frequently avoided
> > in production environments due to the risk of watchdog timeouts induced
> > by synchronous hardware console emission.
> 
> There seems to be various motivations to set/clear
> crash_kexec_post_notifiers.
> 
> Guilherme wanted to add some filtering because some notifiers
> were failing, see
> see https://lore.kernel.org/all/20220108153451.195121-1-gpiccoli@xxxxxxxxxx/
> 
> I believe that they are called after kdump by default because
> the information provided by them is included in the dump.
> This idea is supported by the commit f06e5153f4ae2e2f3
> ("kernel/panic.c: add "crash_kexec_post_notifiers" option
> for kdump after panic_notifers").
> 
> On the other hand, crash_kexec_post_notifiers is explicitely when
> the kernel is running on some hypervisors because the hypervisors
> need to get notified about the panic() before crash dump.
> 
> All I want to say is that the situation around
> crash_kexec_post_notifiers is much more complicated. And I hear
> about the watchdog timeouts in this context for the first time.

Hi Petr,

Thank you for your feedback.

Interesting. Guilherme's work on classifying and filtering panic notifiers
sought to address a well-known vulnerability: namely, that third-party or
device driver callbacks registered on panic_notifier_list can be fragile,
corrupt state, or hang the crashed kernel before kdump has had an
opportunity to boot.

However, my proposal here is quite distinct from Guilherme's effort, though
it shares the goal of making pre-kdump execution dependable.

    1.  Bypassing panic notifier entirely

        I deliberately avoid invoking panic_notifier_list. The instability
        of arbitrary notifier callbacks that Guilherme highlighted is
        precisely why some operators avoid crash_kexec_post_notifiers in
        production. Our intent is strictly to collect well-defined kernel
        state via sys_info()

    2.  Purely in-memory collection

        This patch addresses both dilemmas simultaneously:
            - It eliminates the risk of notifier instability by not
              invoking panic_notifier_list.

            - It eliminates console stalls by suppressing console emission
              and freezing the printk tail, recording the telemetry purely
              into the printk ring buffer in RAM (within milliseconds)
              before jumping to __crash_kexec().

Consequently, the capture kernel receives an enriched log_buf (extractable
via vmcore-dmesg.txt or the vmcore) with virtually zero risk of hardware
watchdog trips or notifier lockups.

> > To resolve this dilemma, introduce the crash_kexec_in_memory_sys_info
> > boot parameter. When enabled, it captures diagnostic telemetry directly
> > into the printk ring buffer entirely in RAM before jumping to
> > __crash_kexec(), completing in milliseconds rather than tens of seconds.
> > 
> > To make this safe, fast, and reliable without risking buffer overflow:
> >     1.  Scoped console flush suppression
> > 
> >         Provide printk_suppress_console_flush(bool) to clear the console
> >         flush mask in printk_get_console_flush_type(). Messages written
> >         via vprintk_store() remain in the printk ring buffer in memory
> >         and avoid synchronous hardware console emission and waking
> >         kthreads.
> 
> This might help when the claim about watchdog reports is true.
> I am not sure about it. Anyway, there are other ways how to
> prevent watchdogs stepping in (touching them, disabling them, ...)

Regarding watchdog intervention, the concern does not stem from internal
kernel watchdogs (e.g. softlockup watchdog), but rather from autonomous
vendor-specific hardware/platform watchdogs (e.g. hpwdt).

A situation occurs with IPMI watchdog
(i.e. drivers/char/ipmi/ipmi_watchdog.c) configured with pre-timeouts,
where the BMC asserts an NMI and independently enforces a hard reset after
a brief remaining window.

In these configurations:
    - "Touching" the watchdog is ineffective i.e. touch_nmi_watchdog() has
      no awareness of external hardware or BMC timers.

    - Some production environments mandate nowayout=1
      (drivers/char/ipmi/ipmi_watchdog.c), which deliberately forbids
      software from stopping the timer.

On a slow serial console or IPMI Serial-over-LAN (e.g. 115200 baud),
flushing comprehensive sys_info() output (e.g. tasks or stacktraces) can
exceed 10 seconds, allowing the hardware timer to expire and reset the
system before kdump can initialise.

Regarding your remark that console output is important when crashdump
fails, I completely agree. That scenario is explicitly handled:

        sys_info(si_mask);
        kmsg_dump_desc(KMSG_DUMP_PANIC, buf);
        __crash_kexec(NULL);
        printk_freeze_tail(false);
        printk_suppress_console_flush(false);

If __crash_kexec() fails or is not armed, suppression is immediately
unmasked before proceeding down the remainder of panic(). As a result,
console_flush_on_panic() flushes the complete buffer, including both the
panic Oops and the newly gathered sys_info telemetry, out to the physical
console as normal.

To clarify, an independent, external hardware controller will forcibly
reboot the machine if the kernel spends too long flushing diagnostic logs
to slow physical serial lines before jumping to __crash_kexec().

> 
> The console output is important when the crashdump fails.
> 
> >      2. Scoped ring buffer tail freezing
> > 
> >         Introduce printk_freeze_tail(bool) in printk_ringbuffer. When
> >         active, desc_push_tail() and data_push_tail() refuse to advance
> >         the tail. If diagnostic logging exhausts available ring buffer
> >         headroom, new records are safely dropped, guaranteeing that the
> >         initial panic Oops, faulting registers, and primary stack trace
> >         are never overwritten.
> 
> This is another questionable feature. The ring buffer would need to be
> super big to hold all messages since the boot. IMHO, servers are
> normally running hundreds of days and the log buffer gets rotated,
> like the user space logs, ...

Apologies for the misunderstanding regarding the lifecycle of the freeze.
Please note, newly proposed printk_freeze_tail() is not active from boot,
nor does it hinder normal buffer rotation while the system is running. It
is activated strictly for the few milliseconds during which sys_info()
executes:

        printk_freeze_tail(true);
        sys_info(si_mask);
        printk_freeze_tail(false);

Prior to this call, the panic Oops, registers, and faulting stack trace
have already been logged. The sole purpose of freezing the tail during
sys_info() is to prevent newly generated telemetry from wrapping the buffer
and overwriting that initial, critical panic Oops if the remaining headroom
is exhausted.

> >     3.  Execution sequence reordering
> > 
> >         Reorder __sys_info() so compact, high-signal subsystems (e.g.
> >         memory) are collected first, leaving high-volume dumps (all CPU
> >         backtraces, full task lists, and ftrace) for last.
> 
> This might make sense. I am just afraid that it might be a personal
> opinion and we might end up with an endless shuffling here.

Fair point. The intention was merely to prioritise compact, high-density
metrics before unbounded dumps (e.g. all-task backtraces). However, if this
introduces unwanted churn, I am entirely content to drop the reordering and
retain the existing sequence.

> My opinion:
> 
> IMHO, it does not make much sense to dump sys_info() before kdump
> and block consoles. The information is lost when kdump fails.
> The information can be extracted from the crashdump when
> kdump succeeds.

The information is not lost if kdump fails, as suppression is immediately
lifted. In an ideal world, the full vmcore (i.e. makedumpfile -d 31) is
always saved intact. In production, however, dump targets may suffer from
storage exhaustion (e.g. accumulated prior vmcores). When this occurs,
writing the multi-gigabyte /proc/vmcore is truncated mid-stream, leaving
the memory dump corrupt or too incomplete to extract dmesg e.g. via crash
or drgn.

Crucially, kdump utilities (e.g. makedumpfile --dump-dmesg) extract
vmcore-dmesg.txt first. Because it requires only a few megabytes, it
reliably succeeds even when saving the full vmcore fails due to ENOSPC.
Capturing sys_info() in-memory beforehand guarantees that vital state
remains preserved in vmcore-dmesg.txt as the sole surviving diagnostic
artefact.

Kind regards,
-- 
Aaron Tomlin