Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Sean Christopherson]

On Mon, Nov 02, 2026, Yosry Ahmed wrote:
> On Mon, Nov 2, 2026 at 3:20 AM Jackie Yu <ufo19890607@xxxxxxxxx> wrote:
> >
> > Hi Sean, Yosry, and KVM folks,
> >
> > Sean, thank you for the reminder. Sorry about the earlier formatting.
> > We will use plain text for future mailing-list replies.
> >
> > Yosry, thanks for clarifying. We understand that by "missing
> > emulation" you meant the INVLPGB handling needed with distinct
> > per-vCPU ASIDs, not a bug in an existing emulation path in our tree.
> >
> > We agree that intercepting each guest INVLPGB and issuing a separate
> > invalidation for every ASID in a large VM may undermine the benefit
> > we are seeking. Using per-VM ASIDs consistently, rather than
> > switching ASID allocation models based on nested usage, sounds like
> > a promising direction. Our production guests do not run nested VMs,
> > so direct execution before nested use would cover our use case.
> >
> > Your suggestion to change the INVLPGB interception policy when
> > nested virtualization is first used is helpful. If EFER.SVME can be
> > set before a guest runs a nested VM, the first VMRUN seems like a
> > more precise trigger. Would KVM catch that VMRUN before entering L2
> > and enable the necessary intercepts across the VM before any vCPU
> > can execute an INVLPGB requiring nested handling?
> 
> Yeah we can do this. For example, we can have a static branch in the
> VMRUN emulation path that flips between a synchronized path that
> enables the INVLPGB intercept, and the normal path. The static branch
> might be an overkill.
> 
> > Could those
> > intercepts then remain enabled for the VM's lifetime, rather than
> > switching back when the guest leaves nested mode?
> 
> Probably? I think we can only switch back when all vCPUs clear
> EFER.SVME. I doubt it's worth optimizing.
> 
> > We realize that interception would still require correct nested instruction
> > handling.
> >
> > For context, our experimental KVM4 configuration used a VM-shared
> > ASID and separate, out-of-tree guest INVLPGB passthrough. The
> > madvise() latency measurements were:
> >
> >   Workers             INVLPGB off      INVLPGB on
> >   64 mean              21.624 ms        12.505 ms
> >   64 P99               32.760 ms        21.518 ms
> >   128 mean             29.553 ms        20.459 ms
> >   128 P99              39.628 ms        28.160 ms
> >
> > These were single sequential A/B runs with 64 allowed CPUs; the
> > 128-worker run was oversubscribed. The results measure madvise()
> > syscall latency, not IPI counts, and do not quantify any additional
> > flushes from vCPU switches on a physical CPU under per-VM ASIDs.
> >
> > We can share the benchmark source and run further comparisons to
> > help evaluate that trade-off. Would the maintainers consider
> > consistent per-VM ASIDs, with direct guest INVLPGB execution before
> > nested use and interception once nested use begins, a direction
> > worth pursuing?
> 
> As I mentioned, I (not the maintainer) have no problem with per-VM
> ASIDs. We probably want to make sure we don't regress workloads due to
> the extra TLB flushes when multiple vCPUs run on the same pCPU tho.

Short answer: no, I don't think per-VM ASIDs is a direction worth pursing.

I actually really like the *idea* of per-VM ASIDs, but in practice they just
don't work well.  Having to flush when running a different vCPU with the same
ASID is generally acceptable for SEV+, because those types of VMs aren't often
CPU overcommited, and they don't support running as nested VMs[1].  Neither of
those holds true for "regular" VMs.

E.g. when running KVM as L1, having to flush on every vCPU migration becomes a
major pain point, because L0 typically has to intercept the flushes.  With per-vCPU
ASIDs, we have to line of sight to eliding those flushes[2].  That should be a
win for KVM-as-L0 as well, as it'll remove any "artificial" TLB misses due to
technically-unnecessary flushing.  CPU overcommited vCPUs will still suffer more
TLB misses due to cross-vCPU TLB pressure, but they won't get hit with a pile of
misses if a vCPU is bouncing between a small set of CPUs (many setups affine vCPUs
to a pool, versus doing a 1:1 pin).

And despite ASIDs being per-VM for SEV+, that's more of a clever hack to avoid
having to add new ISA to manage encryption keys than anything else.  Maybe AMD
intended ASIDs to be per-VM, but the reality is they can't truly be per-VM since
there is no separate tag and invalidation mechanism for host-controlled mappings.
I.e. the ASID is basically VMX's EPT root and VPID rolled into one, and as a result,
it takes on the properties of both a per-VM thing and a per-vCPU thing.

INVLPGB itself also has problems.  It's a broadcast to the *entire* system.  Even
if there's no work to be done, *all* other pCPUs still need to respond.  I haven't
done benchmarking, but I wouldn't be at all surprised if there are serious noisy
neighbor issues if a misbehaving guest is spamming INVLPGB in a tight loop.  E.g.
Maxim measured the impact of ringing the AVIC doorbell in a tight loop, and found
he could degrade CPU througput by something like 40-50%.

And then if the guest does something that forces KVM to intercept INVLPGB to do
the right thing, I suspect Yosry is right and performance will tank.

So, outside of very specific setups, per-VM ASIDs and INVLPGB passthrough seem
like a lot of complexity for little-to-no benefit.  Add in that per-VM ASIDs will
likely do more harm than good for most other use cases, and I'm not inclinded to
take on maintaining per-VM ASIDs in upstream.

Sorry :-/

[1] It's actually possible to get KVM functional when running as an SEV guest,
    but it's hilariously stupid and completely defeats the value added by SEV.

[2] https://lore.kernel.org/all/aHWjPSIdp5B-2UBl@xxxxxxxxxx

> Other than that, I don't want to rework my series to go back to per-VM
> ASIDs. So if Sean and Paolo are okay with per-VM ASIDs, I think this
> should be done as a follow up work on top. Just my 2c.
>