improving crash dump discussion

[Pavel Tatashin]

I would like to start a discussion about how we can improve Linux
crash dump facility, and use warm reboot  / firmware assistance in
order to more reliably collect crash dumps while using fewer
memory resources and being more performant.

Currently, the main way to collect crash dumps on Linux is to use
kdump.  Kdump uses kexec in order to collect dumps. Kdump makes
use of kexec, which is mature and portable (does not depend on
firmware), but using kexec is not ideal.

I will list some problems with kexec/kdump, and then discuss how some
of them (hopefully most) can be addressed.

1. Expecting a crashing kernel to do the right thing: properly quiesce
devices, CPUs and prepare the machine for the new kernel.

The amount of code that is executed to perform crash kexec reboot is
not trivial. Unfortunately, since we are panicking we already lost
control at some point and the goal would be to reduce the amount of
code executed by the panic handler in order to be able to reliably
collect dumps. There are some ways to improve the reliability of crash
kexec reboot. For example, passing maxcpus=1 kernel parameter is now
the required on almost all platforms, which, unfortunately, has the
downside of forcing crash kernel to use only a single thread to save
core, and thus "makedumpfile --num-thread" is useless if used from
crash kernel.

2. Unlike booting from firmware, the PCI, CPUs, interrupt controllers,
DMAs mappings, and I/O devices are not reinitialized and might not be
in a consistent state.

The reset_devices, irqpoll, and other kernel parameters also intend to
mitigate these shortfalls by requiring drivers to do the resetting
themselves. Also, the kernel is usually smart enough to ignore
spurious interrupts, but this is fragile.

3. There is a blackout window during boot where collecting a crash dump
is not possible.

With current kdump it is possible to collect crashes that occur after
the kernel early boot is finished. During early boot we do a lot:
determine platform, initialize mm, initialize clock, scheduler, and
start other CPUs. Only after entering usermode, we are able to kexec
load crash kernel into memory after which crash can be collected.

4. Kdump is not compatible with hardware watchdog resets

When a hardware watchdog causes a reset, software is not involved, and
therefore we lose the entire machine state.

5. Crash kernel requires memory reservation

Crash kernel can't use the memory that was used by the crashing
kernel, therefore memory must always be reserved that is wasted during
normal operation, and only contains the image of the crash kernel.

6. Crash kernel requires special image and two reboots

Special crash image is usually required to reduce the number of loaded
modules, and also to reduce the system to the bare minimum so that it
can be booted in the small reserved space. Also, after the crash
kernel collects the core dump, we reboot back to the normal kernel,
thus two reboots are needed in order to recover after the crash.

==========================================================================

On the other hand, powerpc can optionally use firmware assisted
kdump (fadump). The benefits of fadump:

1. reboot through firmware happens, and thus all devices are reset to
their initial state

2. memory for the crash kernel does not need to be reserved if CMA is
used and user pages do not need to be preserved (commonly there is no
need to preserve user pages to debug kernel panics).

3. fadump crash format is identical to kdump (ELF /proc/vmcore),
therefore tools are the same, i.e. crash(8), makedumpfile, and other
all can be used.

4. No need to have a special crash kernel image and no need to do a
second reboot from the crash kernel.

The following services are expected from firmware in order for fadump to work:

1. Ability to do warm reboot

Preserve memory content across reboot. Firmware must not zero
(initialize) memory content. From my experience, this is actually
common nowadays: I see this happens on my AMD desktop with x570 chip +
UEFI BIOS, we do this at Microsoft both on larger Xeon servers with
UEFI firmware, and on small arm64 devices which use device trees instead
of EFI for performance reasons, and also to preserve emulated pmem
devices across reboot. We also did it at Oracle on SPARC sun4v machines
where sun4v hypervisor would not reset memory content on every reboot for
performance reasons.

2. Ability to register preserved memory region with firmware

The first kernel uses firmware to reserve a region of memory that must
be preserved when rebooted. Firmware and bootloader must not allocate
from preserved regions.

3. Ability to copy boot memory source to destination.

On powerpc, boot must start from a lower address, similar like on x86.
Also, boot memory is a region of memory that can be used by the kernel
to boot, and the rest is added later once the kernel decides to
unreserve it: i.e. after vmcore is saved.

The copy boot memory is not strictly necessary: the panicking kernel
can do the copy on platforms where boot must start from a lower
address, and on other platforms where boot can be done from any
address the copy is not needed at all (i.e. ARM64, x64).

What it comes down to is that there is little that firmware needs to
do in order to help Linux to do a more reliable crash dump. It must
provide an ability by the kernel to reserve a region of memory from
which firmware/bootloader won’t do allocations, and optionally on
platforms where the kernel must always boot from a predefined physical
address firmware should be able to copy boot memory content. The rest
can be done by the kernel alone.

Support for hardware watchdog resets is a little more complicated as
it would involve firmware to copy CPUs registers content to a
predefined place, but it should also be achievable.

We could agree on an interface that the kernel would support for both
EFI based firmware and device-tree based firmware. We could also add
this support to open source projects such as linuxboot, coreboot, OVMF
type of firmware and to boot loaders u-boot, grub.

Pasha