Re: Some options abort RFC: KVM: SVM: Rework ASID management

[Jackie Yu]

Hi Sean, Yosry, and KVM folks,

Sean, thank you for the reminder. Sorry about the earlier formatting.
We will use plain text for future mailing-list replies.

Yosry, thanks for clarifying. We understand that by "missing
emulation" you meant the INVLPGB handling needed with distinct
per-vCPU ASIDs, not a bug in an existing emulation path in our tree.

We agree that intercepting each guest INVLPGB and issuing a separate
invalidation for every ASID in a large VM may undermine the benefit
we are seeking. Using per-VM ASIDs consistently, rather than
switching ASID allocation models based on nested usage, sounds like
a promising direction. Our production guests do not run nested VMs,
so direct execution before nested use would cover our use case.

Your suggestion to change the INVLPGB interception policy when
nested virtualization is first used is helpful. If EFER.SVME can be
set before a guest runs a nested VM, the first VMRUN seems like a
more precise trigger. Would KVM catch that VMRUN before entering L2
and enable the necessary intercepts across the VM before any vCPU
can execute an INVLPGB requiring nested handling? Could those
intercepts then remain enabled for the VM's lifetime, rather than
switching back when the guest leaves nested mode? We realize that
interception would still require correct nested instruction handling.

For context, our experimental KVM4 configuration used a VM-shared
ASID and separate, out-of-tree guest INVLPGB passthrough. The
madvise() latency measurements were:

  Workers             INVLPGB off      INVLPGB on
  64 mean              21.624 ms        12.505 ms
  64 P99               32.760 ms        21.518 ms
  128 mean             29.553 ms        20.459 ms
  128 P99              39.628 ms        28.160 ms

These were single sequential A/B runs with 64 allowed CPUs; the
128-worker run was oversubscribed. The results measure madvise()
syscall latency, not IPI counts, and do not quantify any additional
flushes from vCPU switches on a physical CPU under per-VM ASIDs.

We can share the benchmark source and run further comparisons to
help evaluate that trade-off. Would the maintainers consider
consistent per-VM ASIDs, with direct guest INVLPGB execution before
nested use and interception once nested use begins, a direction
worth pursuing?

Best regards,


Sean Christopherson <seanjc@xxxxxxxxxx> 于2026年10月30日周五 00:09写道：
>
> Please send mails as plain text, otherwise they get ignored by the mailing lists
> and don't get archived by lore.
>
> On Thu, Oct 29, 2026, Jackie Yu wrote:
> > Subject: Re: [PATCH v2 00/29] KVM: nSVM: Optimize nSVM TLB flushes
> >
> > Hi Yosry,
> >
> > Thank you for the quick response. Two clarifications from our side:
> >
> > 1. About the boot failures: no emulation is involved in our
> > experiment. Our out-of-tree changes enable direct guest
> > INVLPGB/TLBSYNC execution by clearing the KVM intercepts, without any
> > software emulation of the instruction. The failures occur with this
> > unintercepted passthrough combined with static per-vCPU ASIDs, so they
> > are not missing cases in an emulation path. Under the per-vCPU design
> > we did not attempt to intercept and emulate INVLPGB at all.
> >
> > 2. We do not use nested virtualization in our deployments. L1 guests
> > in our production scenarios run without nested guests. Therefore, the
> > ASID-replacement behavior you quoted from the APM does not affect our
> > current use case: with one shared ASID per non-nested VM, an
> > unintercepted guest INVLPGB would be scoped to that VM's ASID, which
> > matches what our per-VM configuration relies on today.
> >
> > Regarding the direction: we understand that with per-vCPU ASIDs,
> > correct handling would require intercepting INVLPGB and re-issuing a
> > flush for every ASID in use by the VM, which you note is unlikely to
> > perform well on large VMs. We also note your point that as long as any
> > vCPU uses nested, KVM must intercept INVLPGB anyway because the
> > hardware replaces the requested ASID with the current ASID.
> >
> > If we read the situation correctly, the practical options seem to be
> > either (a) a per-VM shared ASID mode restricted to non-nested VMs,
> > with unintercepted guest INVLPGB, plus mandatory interception whenever
> > nested is in use, or (b) always intercepting and virtualizing INVLPGB
> > under per-vCPU ASIDs. Either way, direct passthrough would at most
> > cover the non-nested case. We would appreciate Sean's and Paolo's
> > input on which direction upstream prefers; we are glad to provide more
> > details of our configuration or run tests that would help decide
> > between these options.
> >
> > Thanks,
> >
> > Jackie
> >
> >
> > Yosry Ahmed <yosry@xxxxxxxxxx> 于2026年10月29日周四 14:08写道：
> >
> > > On Wed, Oct 28, 2026 at 1:15 PM Yosry Ahmed <yosry@xxxxxxxxxx> wrote:
> > > >
> > > > On Wed, Oct 28, 2026 at 6:37 AM Jackie Yu <ufo19890607@xxxxxxxxx> wrote:
> > > > >
> > > > > Hi, guys!
> > > > >
> > > > > Jackie Yu <ufo19890607@xxxxxxxxx> 于2026年10月28日周三 21:36写道：
> > > > >>
> > > > >> Hi Yosry and KVM folks,
> > > > >>
> > > > >> We would appreciate your guidance on supporting INVLPGB in ordinary
> > > (non-SEV)
> > > > >> L1 guests alongside the proposed static per-vCPU ASIDs.
> > > > >>
> > > > >> The 2025 ASID rework RFC proposed one static ASID per L1 VM and listed
> > > > >> "enabling INVLPGB virtualization" among its motivations [1]. The June
> > > > >> 2026 RFC v2 changed the design from per-VM to per-vCPU ASIDs [2]. In
> > > > >> the October v2 series, patches 2-11 establish static per-vCPU ASIDs,
> > > > >> while the remaining patches primarily address nested TLB flushes [3].
> > > > >> Our question is how guest INVLPGB should be supported under the
> > > > >> per-vCPU design.
> > > > >>
> > > > >> Some of our workloads frequently discard and repopulate memory using
> > > > >> patterns involving madvise() and memset(). Broadcast IPI-based TLB
> > > > >> shootdowns are expensive for these workloads, so we are interested in
> > > > >> making AMD INVLPGB available to guests. We use a small benchmark to
> > > > >> approximate the resulting shared-address-space pressure:
> > > > >>
> > > > >>     region = mmap(256 MiB, MAP_SHARED | MAP_ANONYMOUS);
> > > > >>     memset(region, 0xab, 256 MiB);       // Initial population
> > > > >>     start N workers, pinned across 64 allowed CPUs;
> > > > >>     each worker repeatedly reads pages in its own slice of region;
> > > > >>
> > > > >>     repeat 500 warmup + 2,000 measured iterations:
> > > > >>         start = clock_gettime(CLOCK_MONOTONIC);
> > > > >>         madvise(region, 256 MiB, MADV_DONTNEED);
> > > > >>         record elapsed madvise() time;
> > > > >>         write one page in every 64 pages to repopulate part of region;
> > > > >>
> > > > >>     stop and join the workers;
> > > > >>
> > > > >> The benchmark uses memset() for initial population; subsequent
> > > > >> iterations re-touch pages rather than repeatedly calling memset().
> > > > >> The numbers below measure the advisor thread's madvise() syscall
> > > > >> latency, not worker throughput.
> > > > >>
> > > > >> In our experimental KVM4 configuration with a VM-shared ASID and our
> > > > >> separate guest INVLPGB passthrough changes, we observed:
> > > > >>
> > > > >>                          INVLPGB off     INVLPGB on    Difference
> > > > >>     64 workers  mean       21.624 ms       12.505 ms     -42.17%
> > > > >>                 P99        32.760 ms       21.518 ms     -34.32%
> > > > >>     128 workers mean       29.553 ms       20.459 ms     -30.77%
> > > > >>                 P99        39.628 ms       28.160 ms     -28.94%
> > > > >>
> > > > >> These are single, sequential A/B runs for each worker count, with
> > > > >> 500 warmup and 2,000 measured calls per run. The process had 64
> > > > >> allowed CPUs, so the 128-worker case was oversubscribed. The
> > > > >> measurements show a useful latency difference, but do not by
> > > > >> themselves establish INVLPGB instruction counts or IPI reductions.
> > > > >>
> > > > >> We then experimentally combined the October per-vCPU ASID changes for
> > > > >> ordinary guests with our existing, out-of-tree L1 INVLPGB/TLBSYNC
> > > > >> passthrough. With guest INVLPGB enabled, a one-vCPU guest boots, but
> > > > >> otherwise comparable two-vCPU and 64-vCPU guests fail early during
> > > > >> boot. Keeping QEMU's +invlpgb setting while adding clearcpuid=419
> > > > >> to the same guest kernel allows the multi-vCPU guest to boot; the
> > > > >> guest logs "Clearing CPUID bits: invlpgb".
> > > >
> > > > I imagine for INVLPGB to work correctly with per-vCPU ASIDs, you need
> > > > to intercept INVLPGB and basically issue an INVLPGB for each ASID used
> > > > by this VM (nr_vcpus or 2 * nr_vcpus with nested). I can't imagine
> > > > this would perform well on large VMs, not sure if virtualizing INVLPGB
> > > > would still make sense.
> > > >
> > > > Regarding the boot failure, I would guess there are some missing cases
> > > > due to the complexity of emulating INVLPGB without a shared per-VM
> > > > ASID.
> > > >
> > > > >> These controlled boot comparisons identify a problem with combining
> > > > >> static per-vCPU ASIDs and direct guest INVLPGB/TLBSYNC passthrough:
> > > > >> the multi-vCPU guest fails to boot when it uses INVLPGB, but boots
> > > > >> when its INVLPGB CPUID bit is cleared. A one-vCPU guest also boots
> > > > >> with INVLPGB enabled.
> > > > >>
> > > > >> In this configuration, an unintercepted guest INVLPGB targets the
> > > > >> invoking vCPU's hardware ASID, while sibling vCPUs have distinct
> > > > >> ASIDs.
> > > >
> > > > It's neat that INVLPGB will automatically use the current ASID and not
> > > > ASID=0 for guest usage.
> > >
> > > Hmm, actually, how is this supposed to work with nested
> > > virtualization? The APM says:
> > >
> > >     A guest that executes a legal INVLPGB that is not intercepted will have
> > >     the requested ASID field replaced by the current ASID and the valid
> > >     ASID bit set before doing the broadcast invalidation.
> > >
> > > If any ASID passed by an L1 guest is replaced by its own ASID, then if
> > > L1 is flushing L2's ASID (i.e. a non-zero ASID), passing through
> > > INVLPGB would do the wrong thing and flush L1's ASID instead, right?
> > >
> > > Seems like if the VM is using nested (on any vCPU), KVM must intercept
> > > INVLPGB anyway to correctly handle the ASID?
> > >
> > > The APM also says:
> > >
> > >     Two forms of INVLPGB intercepts, conditional and unconditional, are
> > >     available to the hypervisor. The unconditional intercept traps all
> > > guest
> > >     usage of INVLPGB. The conditional intercept traps only
> > >     illegally-specified INVLPGB instructions.
> > >
> > > When I read "conditional", I was really hoping it was conditional on
> > > the ASID being zero or non-zero, as KVM really only needs to intercept
> > > INVLPGB with non-zero ASIDs (assuming the ASID is shared to begin
> > > with).
> > >
> > > Am I missing something?
> > >