Re: [PATCH v4 00/15] Introducing AMD x2AVIC and hybrid-AVIC modes

[Maxim Levitsky]

On Sat, 2022-05-07 at 21:39 -0500, Suravee Suthikulpanit wrote:
> Introducing support for AMD x2APIC virtualization. This feature is
> indicated by the CPUID Fn8000_000A EDX[14], and it can be activated
> by setting bit 31 (enable AVIC) and bit 30 (x2APIC mode) of VMCB
> offset 60h.
> 
> With x2AVIC support, the guest local APIC can be fully virtualized in
> both xAPIC and x2APIC modes, and the mode can be changed during runtime.
> For example, when AVIC is enabled, the hypervisor set VMCB bit 31
> to activate AVIC for each vCPU. Then, it keeps track of each vCPU's
> APIC mode, and updates VMCB bit 30 to enable/disable x2APIC
> virtualization mode accordingly.
> 
> Besides setting bit VMCB bit 30 and 31, for x2AVIC, kvm_amd driver needs
> to disable interception for the x2APIC MSR range to allow AVIC hardware
> to virtualize register accesses.
> 
> This series also introduce a partial APIC virtualization (hybrid-AVIC)
> mode, where APIC register accesses are trapped (i.e. not virtualized
> by hardware), but leverage AVIC doorbell for interrupt injection.
> This eliminates need to disable x2APIC in the guest on system without
> x2AVIC support. (Note: suggested by Maxim)
> 
> Regards,
> Suravee
> 
> Testing for v4:
>   * Tested booting a Linux VM with x2APIC physical and logical modes upto 512 vCPUs.
>   * Test enable AVIC in L0 with xAPIC and x2AVIC modes in L1 and launch L2 guest
>   * Test partial AVIC mode by launching a VM with x2APIC mode
> 
> Changes from v3:
> (https://lore.kernel.org/lkml/ff67344c0efe06d1422aa84e56738a0812c69bfc.camel@xxxxxxxxxx/T/)
>  * Patch  3 : Update logic force_avic
>  * Patch  8 : Move logic for handling APIC disable to common code (new)
>  * Patch  9 : Only call avic_refresh_apicv_exec_ctrl
>  * Patch 12 : Remove APICV_INHIBIT_REASON_X2APIC, and add more comment for hybrid-AVIC mode
> 
> Suravee Suthikulpanit (15):
>   x86/cpufeatures: Introduce x2AVIC CPUID bit
>   KVM: x86: lapic: Rename [GET/SET]_APIC_DEST_FIELD to
>     [GET/SET]_XAPIC_DEST_FIELD
>   KVM: SVM: Detect X2APIC virtualization (x2AVIC) support
>   KVM: SVM: Update max number of vCPUs supported for x2AVIC mode
>   KVM: SVM: Update avic_kick_target_vcpus to support 32-bit APIC ID
>   KVM: SVM: Do not support updating APIC ID when in x2APIC mode
>   KVM: SVM: Adding support for configuring x2APIC MSRs interception
>   KVM: x86: Deactivate APICv on vCPU with APIC disabled
>   KVM: SVM: Refresh AVIC configuration when changing APIC mode
>   KVM: SVM: Introduce helper functions to (de)activate AVIC and x2AVIC
>   KVM: SVM: Do not throw warning when calling avic_vcpu_load on a
>     running vcpu
>   KVM: SVM: Introduce hybrid-AVIC mode
>   KVM: x86: Warning APICv inconsistency only when vcpu APIC mode is
>     valid
>   KVM: SVM: Use target APIC ID to complete x2AVIC IRQs when possible
>   KVM: SVM: Add AVIC doorbell tracepoint
> 
>  arch/x86/hyperv/hv_apic.c          |   2 +-
>  arch/x86/include/asm/apicdef.h     |   4 +-
>  arch/x86/include/asm/cpufeatures.h |   1 +
>  arch/x86/include/asm/kvm_host.h    |   1 -
>  arch/x86/include/asm/svm.h         |  21 +++-
>  arch/x86/kernel/apic/apic.c        |   2 +-
>  arch/x86/kernel/apic/ipi.c         |   2 +-
>  arch/x86/kvm/lapic.c               |   6 +-
>  arch/x86/kvm/svm/avic.c            | 191 ++++++++++++++++++++++++++---
>  arch/x86/kvm/svm/svm.c             |  56 +++++----
>  arch/x86/kvm/svm/svm.h             |   6 +-
>  arch/x86/kvm/trace.h               |  18 +++
>  arch/x86/kvm/x86.c                 |   8 +-
>  13 files changed, 262 insertions(+), 56 deletions(-)
> 

Patch series looks good.

I will smoke test it today on my normal AVIC, just in case.

Did you had a chance to look at my comments on your report
that nesting got broken by my nested PAUSE filtering patch?

I tried to reproduce it on my side, so far no luck.

I tried to oversubscribe L1, by booting a VM with 16 vCPUs
all pinned to single physical CPU, and then booting a nested guest 
in it with about the same amount of vCPUs. Slow but it did work.


Also did you had a chance to look for my comments about the AMD's manual
asking the user to flush guest's TLB when changing apic backing page,
regardless of ASID?

Best regards,
	Maxim Levitsky