[Hypervisor Live Update] Notes from November 2, 2026

[Pasha Tatashin]

Hi everybody,

Here are the notes from our Hypervisor Live Update call held on November 2.
These notes are intended to bring those who could not attend up to speed
and to keep the momentum going between meetings.

----->o-----
Orphaned VMs & Linux Caretaker

* Shift to Specialized Full Linux Caretaker: Evolving the Orphaned VMs RFC
  architecture presented at LPC:

  https://lore.kernel.org/all/20260920193650.3373435-1-pasha.tatashin@xxxxxxxxxx

  Pasha moved from a standalone KVM-only caretaker runtime to a specialized
  full Linux caretaker kernel. Using Linux directly avoids reimplementing and
  abstracting KVM execution logic in a separate environment and enables
  handling VMM-level VM exits (including select virtio exits such as pvpanic).
  To stay lightweight and isolated, the caretaker boots strictly within its
  assigned memory and physical CPUs—skipping PCI discovery and device access
  entirely and leaving the global KHO tree owned by the host machine owner.

* Supporting Upstream Series:
  - CPU Hotplug Performance Improvements: Standalone series accelerating CPU
    offlining and onlining to reduce physical CPU transfer blackout time
    between the host and caretaker domains:
    https://lore.kernel.org/all/20261019011446.834487-1-pasha.tatashin@xxxxxxxxxx
  - Read-Only KHO: Introduces a read-only KHO mode for the caretaker kernel
    (currently awaiting upstream review):
    https://lore.kernel.org/all/20261025174017.3870481-1-pasha.tatashin@xxxxxxxxxx
    Following discussions with Pratyush Yadav and Mike Rapoport, Pasha will
    continue enforcing that KHO-preserved memory outside the caretaker's own
    RAM is excluded from its direct map and mapped read-only on demand,
    preventing accidental KHO modification and avoiding unscalable direct-map
    / struct page exceptions. Read-only memfd changes (previously prototyped
    solely to exercise an end-to-end selftest) will stay separate while core
    read-only KHO is reviewed.
  - Early Session Freeze (LIVEUPDATE_SESSION_FREEZE): v4 is posted upstream:
    https://lore.kernel.org/all/20261026014948.162559-1-pasha.tatashin@xxxxxxxxxx/

* Caretaker Handover Lifecycle & Session Pinning: Pasha walked through the
  caretaker lifecycle: a global session (e.g., session_lc) holds the
  caretaker's preserved physical CPUs and boot RAM memfd. On the outgoing
  path, the VMM suspends the VM, freezes its LUO session (resolving circular
  FD dependencies and committing metadata to KHO), and tells the caretaker to
  look up the session by name; on the incoming path after kexec, userspace
  stops the caretaker before retrieving and resuming the VM session. Pasha
  discussed adding an optional LIVEUPDATE_SESSION_FREEZE_COMMIT flag (avoiding
  "_IRREVOCABLE" in case an explicit LIVEUPDATE_CMD_SESSION_UNFREEZE ioctl is
  added later) so frozen sessions remain committed in LUO if the VMM exits.
  Pratyush Yadav, David Matlack, and Samiullah Khawaja noted that session
  protection is symmetric across outgoing and incoming paths: while the
  caretaker is actively using a session's preserved resources, the kernel
  should pin / hold a reference in LUO metadata so userspace cannot unfreeze,
  close, or prematurely retrieve and release backing memory (e.g., memfd
  pages) until the caretaker stops.

* Session Isolation vs. Global KHO Tree: Pratyush and David Matlack also
  asked whether the caretaker needs visibility into the global KHO radix tree
  (whose primary purpose is excluding preserved memory from early boot
  allocations, whereas the caretaker boots from dedicated RAM) or could be
  handed only its target LUO session payload(s) for stronger isolation. Pasha
  noted that the caretaker currently maps KHO nodes read-only only along the
  lookup path at retrieve time, and will evaluate whether KHO tree access is
  still needed if session physical addresses are passed directly.

----->o-----
Cross-Subsystem Versioning & Compatibility Strategy

* Expandable Serialization Header (kho_header_ser) v4: Tarun Sahu has v4 of
  the minimal expandable serialization header series ready to post, adding
  entry_size alongside feature flags per Pratyush's review feedback and
  updating existing KHO/LUO serialized structures across upstream Linux.
  Recording entry_size in the header is necessary for early-boot array
  deserialization (such as luo_session_header_ser entries), where callers do
  not inherently know the entry size written by the previous kernel.

* Header Naming Alignment: Tarun asked whether kho_header_ser should be
  renamed to include "feature" to distinguish it when prepended to
  kho_block_header_ser. Pasha and David Matlack agreed to keep kho_header_ser:
  every serialized structure starts with kho_header_ser, while KHO blocks
  naturally follow it with their block-specific kho_block_header_ser.

* Custom Preserve/Retrieve Arguments & Version UAPI: David Matlack is working
  on custom preserve/retrieve argument support alongside tmpfs preservation.
  Pasha suggested landing the serialization header series first and starting
  upstream discussion on the kernel version UAPI in the coming weeks.

----->o-----
Guest Memory (guest_memfd), KVM & Confidential VMs

* End-to-End KVM & Confidential VM PoC Series: Tarun Sahu prepared a combined
  PoC patch series to demonstrate the end-to-end preservation ABI and future
  extensibility for KVM maintainers, structured in layered stages with
  kselftests for each:
  - Non-confidential VM preservation (guest_memfd, KVM VM FD, and vCPU FD
    preservation + selftests).
  - Intel TDX preservation (feature header + selftests, thoroughly reviewed
    with Sagi Shahar).
  - AMD SEV-SNP and ARM CCA preservation (feature headers + selftests,
    prototyped/LLM-tested to illustrate cross-architecture ABI coverage and
    noted accordingly in the cover letter).

* Rebasing & Split Plan: Tarun is cleaning up the PoC series and rebasing it
  onto kho_header_ser v4. Once kho_header_ser lands in liveupdate/next, he
  will post the PoC (marked not-for-merge) to anchor ABI review, with
  standalone vCPU and guest_memfd preservation series prioritized next.

----->o-----
TDX Module & VM Preservation

* KVM PoC Integration & HKID Preservation: Sagi Shahar confirmed that TDX
  preservation is integrated into Tarun's combined KVM PoC patch set. Ongoing
  work is focused on resolving Intel TDX Host Key ID (HKID) preservation
  across kexec.

----->o-----
PCIe Persistence

* Merging PCI v10 & Sequencing for Linux 7.5: With Linux 7.4-rc1 released on
  November 1, liveupdate/next is now open for Linux 7.5. Rather than delaying
  David Matlack's reviewed PCI Live Update v10 series or posting a PCI v11
  that would require re-collecting Reviewed-by tags across the series, Pasha,
  David, and Tarun agreed on a two-step landing sequence:
  1. Pasha will merge PCI v10 into liveupdate/next immediately so it begins
     soaking in linux-next.
  2. Tarun will base kho_header_ser v4 on liveupdate/next and include a patch
     converting PCI's serialized structures (alongside KHO, LUO, and memfd)
     to kho_header_ser, requiring review only on that conversion patch.

----->o-----
VFIO Persistence

* VFIO Live Update v6 Posted: Vipin Sharma returned from leave and posted v6
  of the VFIO live update series (based on PCI v10). Moving beyond the LPC
  prototype (memmap + DSA), v6 uses preserved memfd as the DMA backing source
  in no-IOMMU mode and has been validated end-to-end across kexec on both
  Intel IGB and DSA devices. Community reviews on v6 are welcome.

* PCI Save State Dependency: To get v6 out for review without stacking another
  unmerged series, Vipin included temporary manual serialization of PCI save
  state into an allocated buffer. David Matlack noted that because the
  kernel's current pci_saved_state layout is code-driven and not ABI-stable
  across kernel versions, his PCI save state rework series must merge ahead
  of VFIO. David will review VFIO v6 while Vipin reviews the PCI save state
  rework series to accelerate landing it first.

----->o-----
IOMMU Persistence & LUO Core

* IOMMU Persistence v6 -> v7: Samiullah Khawaja received Reviewed-by tags
  from Baolu Lu on the Intel IOMMU driver, IOMMU core, and IOMMUFD changes in
  v6, and plans to post v7 soon.

* Preventing memfd Hole-Punching Hazards for IOMMU Mappings: Samiullah
  started an upstream discussion (mm/memfd: add F_SEAL_PUNCH_HOLE) to prevent
  a gap where userspace maps memfd pages into IOMMUFD and punches a hole
  (FALLOC_FL_PUNCH_HOLE) prior to memfd preservation—discarding pinned pages
  that existing seals do not block so they are missing after kexec. Per
  feedback from David Hildenbrand, rather than introducing a new seal UAPI,
  memfd will set a sticky flag whenever a hole is punched, and IOMMU preserve
  will reject memfds with that flag set (leaving standalone memfd preserve
  unchanged). Investigating this path also uncovered and fixed a bug in
  memfd_pin_folios().

* Hitless Replacement & IOTLB Conflicts: Upstream discussions continue on
  hitless IOMMU translation replacement across kexec and evaluating potential
  IOTLB conflicts.

----->o-----
Agenda Rotation for Next Meeting

* Deferred Topics First: Because the first seven agenda items filled the
  hour, Pasha will continue rotating the agenda so that the remaining topics—
  tmpfs File Preservation, KHO & Boot Memory (bootmem), PowerPC & LoongArch
  KHO Support, and Resource Accounting & Session Handover—are covered at the
  beginning of the next call.

----->o-----
Action Items & Next Steps

Pasha:
* Merge PCI Live Update v10 into liveupdate/next for Linux 7.5.
* Post the next revision of the early session freeze series incorporating
  caretaker session pinning / lifecycle updates.
* Evaluate whether the Linux caretaker still requires KHO tree access if
  session payload physical addresses are passed directly.
* Rotate the deferred topics (tmpfs, KHO/bootmem, PowerPC & LoongArch) to the
  top of the agenda for the next meeting.

Tarun:
* Post kho_header_ser v4 (with entry_size) rebased onto liveupdate/next,
  converting PCI v10 alongside existing KHO, LUO, and memfd structures.
* Clean up, rebase, and post the combined KVM + guest_memfd + confidential VM
  (TDX, SEV-SNP, ARM CCA) preservation PoC series once kho_header_ser is in
  liveupdate/next.

David M:
* Review Vipin's VFIO live update v6 series.
* Continue work on generic/custom LUO preserve & retrieve arguments alongside
  the next tmpfs preservation series.

Vipin:
* Review David Matlack's PCI save state rework series so VFIO preservation
  can rebase onto ABI-stable PCI save state structures.

Samiullah:
* Post IOMMU persistence v7 incorporating review feedback and tags.
* Implement the sticky memfd punch-hole tracking flag check for IOMMU
  preserve and continue hitless IOTLB replacement discussions.

Pratyush:
* Review Pasha's read-only KHO and updated early session freeze series, as
  well as Tarun's kho_header_ser v4 series.

Sagi:
* Continue working on TDX HKID preservation across kexec with Tarun's KVM/TDX
  PoC series.

The Group:
* Review Pasha's read-only KHO series and Vipin's VFIO live update v6 series
  on the mailing list.

----->o-----

Next meeting will be on Monday, November 16 at 8am PST (UTC-8), everybody is
welcome!

Join Here: https://meet.google.com/rjn-dmzu-hgq

Please let me know if you'd like to propose additional topics for
discussion, thank you!

Best regards,
Pasha