[ANNOUNCE] kmorph: rebootless kernel crash recovery and live update

[Cong Wang]

Hi all,

I would like to announce kmorph, a tool built on multikernel, the
kernel work that runs several Linux kernels side by side on one
machine. kmorph uses it to let a machine survive a kernel crash, and
upgrade its kernel, without rebooting at all:

    kmorph:      https://github.com/multikernel/kmorph
    multikernel: https://github.com/multikernel/linux

The problem
===========

Today, when a production kernel panics, the best we have is kdump, and
kdump costs two reboots:

  1. The crashed kernel kexecs into a capture kernel, which has to boot
     from scratch, bring up a small user space to save the vmcore.
  2. The machine then reboots again, usually through the firmware, into
     the production kernel, which boots from scratch once more and
     restarts every service.

On a large server that is minutes of downtime per crash, and the
capture kernel sits idle in reserved memory the rest of the time,
doing nothing until the moment it is needed.

What kmorph does
================

With multikernel, each kernel owns its own CPUs, memory and devices,
and the kernels talk to each other over vsock. kmorph uses this to
replace both reboots with a takeover. A second kernel, the successor,
is booted ahead of time on a CPU or two and a slice of memory beside
the running kernel, the predecessor, and watches it. When the
predecessor panics or hangs, the successor fences its CPUs, adopts and
resets its devices, restores the network identity, optionally saves a
kdump compatible vmcore, then hot-adds the remaining CPUs and memory
and becomes the machine's kernel. Detection to a running successor
takes about one second, with no reboot, no firmware and no operator.

The same takeover gives a live kernel update: "kmorph upgrade" hands a
healthy machine over to a successor running a newer kernel. Either way
the new kernel can arm a successor of its own, generation after
generation.

All of the novel work is in the multikernel kernel: fencing, device
and IOMMU handover, and CPU and memory hot-add from a host device tree.
kmorph is plain C orchestration on top of its interfaces, and
integrates with distros the way kdump does, through dracut,
initramfs-tools, kernel-install and a systemd service. It works today
on both x86_64 and ARM64, and RISC-V support is in progress.

nginx back within 2 seconds
===========================

Restarting user space is usually what dominates recovery time, so
kmorph can start selected services inside the successor while it is
still on standby:

    standby = nginx.service

nginx then runs in the successor before anything goes wrong, with its
ports already bound to the machine's addresses. When the predecessor
panics, the successor takes over the NIC, restores the addresses,
switches the root onto the real disk with pivot_root and lets systemd
bring the rest of the machine up around nginx, which was never
stopped. In our testing nginx is serving requests again within 2
seconds of the kernel crash, compared with minutes for a kdump cycle.

Two seconds is where we are today, not where we want to stop. Our
goal is zero downtime: a kernel crash that the services on the machine,
and the clients talking to them, never notice.

How it differs from KHO and LUO
===============================

Kexec HandOver (KHO) and the Live Update Orchestrator (LUO) solve a
related problem, with three key differences:

  - KHO and LUO still go through a kexec reboot, so the new kernel's
    boot is on the critical path. kmorph's successor is already up and
    running before the handover starts.

  - KHO and LUO need a healthy old kernel to serialize its state, so
    they cannot help on a crash. kmorph handles a crash and a planned
    upgrade with the same takeover.

  - KHO and LUO preserve only what a participating subsystem
    explicitly serializes before the kexec. With kmorph, memory is
    preserved by default, because multikernel hot-removes memory
    together with its struct pages.

Join us
=======

This is still early, and we would very much like the community to help
shape it. Feedback of any kind is welcome: on the design, on the kernel
interfaces underneath, on the takeover sequence, on what it would take
to make this useful on your machines, and on the workloads and hardware
we have not thought about yet. Bug reports, testing on real hardware,
patches and reviews are all highly appreciated.


Thanks,
Cong Wang