Re: [RFC PATCH v2 02/25] KVM: SVM: Passthrough the number of supported ASIDs

[Yosry Ahmed]

On Tue, Jul 14, 2026 at 2:28 PM Jim Mattson <jmattson@xxxxxxxxxx> wrote:
>
> On Tue, Jul 14, 2026 at 6:20 AM Yosry Ahmed <yosry@xxxxxxxxxx> wrote:
> >
> > On Mon, Jun 15, 2026 at 5:42 PM Yosry Ahmed <yosry@xxxxxxxxxx> wrote:
> > >
> > > KVM currently hardcodes the number of supported ASIDs in CPUID to 8. A
> > > KVM guest (L1) would then do a full TLB flush (i.e.
> > > TLB_CONTROL_FLUSH_ALL_ASID) every time it runs out of ASIDs on a vCPU
> > > and updates the generation (see new_asid()).
> > >
> > > This is currently harmless, as KVM (L0) uses the same ASID for both L1
> > > and L2, and flushes that ASID on nested transitions. However, following
> > > changes will add proper ASID emulation and a separate ASID for L2,
> > > minimizing the TLB flushes on nested transitions. At that point, a full
> > > TLB flush from a KVM guest (L1) would flush both L1 and L2 ASIDs, so
> > > should be avoided as much as possible.
> > >
> > > Passthrough the number of ASIDs in hardware instead of hardcoding 8, to
> > > reduce the chances of an L1 guest flushing its own TLB entries
> > > unnecessarily on a nested VMRUN.
> > >
> > > In practice, there is no harm in exposing a large number of ASIDs to the
> > > guest, even larger than what hardware supports, as KVM never actually
> > > uses the value of ASID from vmcb12. Even with a separate L2 ASID, KVM
> > > would allocate a (supported) ASID for L2, and just flush that same ASID
> > > every time L1 changes the ASID in vmcb12.
> > >
> > > That being said, avoid the temptation of just advertising the maximum
> > > possible number of ASIDs (i.e. 0xFFFFFFFF), in case any peculiar guest
> > > OS does not handle that properly.
> > >
> > > Note: QEMU currently hardcodes the number of ASIDs to 16, so this change
> > > doesn't help QEMU VMs (without making a similar change in QEMU).
> >
> > I had a discussion with Jim internally about what our VMM should
> > advertise as the number of ASIDs, and based on that I think passing
> > through the hardware value may not be the best idea, at least not
> > without some disclaimers.
> >
> > Jim pointed out that AMD CPUs will ignore unsupported ASID bits, or at
> > least earlier ones did, even though it's not mentioned in the APM
> > (last time I checked). This means that technically KVM can ignore the
> > higher bits of the ASID set by L1 beyond supported ones (e.g. if it
> > uses a hashtable to multiplex L1 ASIDs onto several physical ASIDs).
> > KVM doesn't currently do so with or without this series, but it could.
> >
> > The problem happens if a migration pool contains multiple AMD CPUs
> > with a different number of ASIDs in hardware. If the VMM advertises
> > the greater value of the two, it would technically be advertising an
> > unsupported number of ASIDs on one of the CPUs. If KVM ignores the
> > upper bits based on what it supports, not what userspace advertises in
> > CPUID, then userspace would be shooting itself in the foot.
> >
> > I think technically speaking userspace would be in the wrong for using
> > a number of ASIDs above what KVM_GET_SUPPORTED_CPUID provides, and it
> > should use a number of ASIDs that is supported by all CPUs in a
> > migration pool. However, this makes the VMM's life difficult, say if
> > AMD decides to reduce the number of ASIDs in a newer CPU just for
> > kicks.
> >
> > All of this is hypothetical in nature, but it's a very valid concern imo.
> >
> > I think there are several ways we can handle this so that VMMs can
> > reasonably start advertising more ASIDs to guests without these
> > complications:
> >
> > 1. We can just advertise a larger ASID value instead of 8 (say 16K or
> > 32K), rather than passthrough what's in hardware. I assume this would
> > mean that KVM is more-or-less committing to supporting that number of
> > ASIDs regardless of underlying hardware (i.e. KVM wouldn't reduce it
> > in the future). In this case, a VMM can just advertise the same value.
> >
> > 2. We can advertise the maximum number of ASIDs (2^32 - 1?), which is
> > the same as option (1), except that we're biting the bullet and
> > preventing the possibility of KVM ignoring upper bits completely.
> > Might be too aggressive.
> >
> > 3. Document (somewhere) that KVM should use the userspace advertised
> > value as the source of truth for ignorable bits, even if KVM
> > technically supports a lower value. As I am writing this I am
> > realizing it doesn't make much sense as it means that the value
> > supported by KVM is meaningless. Maybe that comes naturally with the
> > "don't break userspace" rule, but it could also fall under the
> > category of userspace shooting itself in the foot and KVM letting it.
> >
> > 4. Do nothing, let this be userspace's problem, and hope that AMD
> > doesn't reduce the number of ASIDs in a future CPU.
> >
> > I think option (1) is probably the most reasonable, we advertise 32K
> > ASIDs, which is what recent AMD CPUs advertise anyway, but it's
> > more-or-less KVM committing to 32K ASIDs even if a newer AMD CPU
> > doesn't support them.
>
> I think Bulldozer (family 15H) first introduced support for 32K ASIDs.
> KVM still supports CPUs older than that, doesn't it?
>
> The tricky part about advertising more ASIDs than the host supports on
> pre-Bulldozer CPUs is that those CPUs don't have FlushByASID. So, do
> you flush all ASIDs when one is reused?

Currently KVM will flush all ASIDs when running out, then start reusing.

With my patches, KVM allocates a single ASID as the fallback ASID, and
keeps reusing that one after running out of ASIDs. Without
FlushByASID, KVM falls back to flushing all ASIDs. Performance will
probably suck, but I don't think we care about performance that much
on these CPUs?