Re: [PATCH v3] RISC-V: KVM: Avoid synchronous IPIs on VMID rollover

[Anup Patel]

On Fri, Sep 11, 2026 at 5:24 PM Wenlong Li <wenlongli486@xxxxxxxxx> wrote:
>
> The current VMID rollover path synchronously invokes HFENCE.GVMA on
> all online CPUs using on_each_cpu_mask(). This sends IPIs to remote
> CPUs and causes a vCPU running in guest mode on a targeted CPU to exit.
> The rollover path then waits for all CPUs to complete their local TLB
> flushes before VMID allocation can continue.
>
> Make VMID rollover lazy to remove this cross-CPU synchronization.
>
> Track active and reserved software VMIDs for each possible CPU. On
> rollover, preserve hardware VMIDs that may still have stale
> translations, clear the active VMID, and mark a local TLB flush
> pending for each CPU.
>
> Do not send IPIs or otherwise force remote vCPUs to exit during
> rollover. Instead, when a CPU next activates a VMID before guest
> entry, perform the pending local HFENCE.GVMA, install the current
> hardware VMID in HGATP, and only then publish the new active VMID.
>
> Use the same software VMID snapshot for both HGATP programming and
> active publication. When switching identities on the fast path, program
> HGATP before cmpxchg. A concurrent rollover clears active, making the
> cmpxchg fail and forcing activation through the locked slow path.
>
> Perform VMID activation after preemption is disabled in the guest
> entry path. This keeps the per-CPU active VMID and pending flush state
> associated with the same CPU that subsequently enters guest mode.
>
> Keep reserved hardware VMIDs unavailable to the bitmap allocator
> while stale translations associated with them may still exist. This
> prevents a hardware VMID from being reused prematurely after a
> generation rollover.
>
> Represent a software VMID as a 64-bit value combining a generation
> with a hardware VMID, independent of XLEN. This keeps the generation
> space large on RV32 while hardware VMIDs and bitmap indices remain
> native unsigned long values. Extract only the hardware VMID when
> programming HGATP or issuing VMID-specific fences.
>
> Reserve software VMID 0 as the unallocated identifier. When the
> generation counter wraps to 0, compute the next nonzero generation
> locally and publish it atomically, so generation 0 is never visible to
> lockless readers.
>
> Reserve hardware VMID 0. Disable VMID allocation when the hardware
> VMID space is too small to retain one reserved VMID per possible CPU
> while still leaving a VMID available for a new allocation.
>
> Invalidate the per-CPU VMID bookkeeping when virtualization is
> disabled, so stale G-stage translations are flushed before the CPU
> next enters a guest.
>
> This removes IPIs and remote CPU synchronization from the VMID
> rollover path, avoiding VM exits caused solely by VMID exhaustion on
> another CPU.
>
> Tested on QEMU with two host CPUs and the VMID width temporarily
> forced to 2 bits. Repeated rollover, protection against premature VMID
> reuse, activation-versus-rollover stress, and vCPU migration were
> exercised with multiple concurrent guests without guest failures or
> host warnings.

NACK.

The synchronous IPI on VMID roll-over is required for correctness so
that all running Guests/VMs immediatly update their VMID.

Regards,
Anup