Re: [PATCH 0/7] KVM: x86: APX reg prep work

[Paolo Bonzini]

On 4/3/26 01:19, Sean Christopherson wrote:
> On Wed, Mar 11, 2026, Paolo Bonzini wrote:
> > On 3/11/26 01:33, Sean Christopherson wrote:
> > > Clean up KVM's register tracking and storage in preparation for landing APX,
> > > which expands the maximum number of GPRs from 16 to 32.
> > >
> > > This is kinda sorta an RFC, as there are some very opinionated changes.  I.e.
> > > if you dislike something, please speak up.
> > >
> > > My thought is to treat R16-R31 as much like other GPRs as possible (though
> > > maybe we don't need to expand regs[] as sketched out in the last patch?).
> >
> > The cleanups in patches 1-4 are nice.
> >
> > For APX specifically, in abstract it's nice to treat R16-R31 as much as
> > possible as regular GPRs.  On the other hand, the extra 16 regs[] entries
> > would be more or less unused, the ugly switch statements wouldn't go away.
>
> Hmm, yeah, but only if XSAVE is the source of truth for guest R16-R31.
>
> Do we know what the compiler and/or kernel rules for using R16-R31 will be?

On the compiler side, they are enabling APX only with new enough -march, 
or with -march=native if the host has APX. This, by the way, implies 
CONFIG_X86_NATIVE_CPU is currently hosed on Diamond Rapids and newer 
machines.

From what I remember of Chang Seok Bae's presentation at Plumbers last 
year, right now there's no plan to have the kernel use APX, except 
possibly through the usual kernel_fpu_begin/end.

> E.g. if C code is allowed to use R16-R31 at will, then KVM will either need to
> swap R16-R31 in assembly, or annotate a pile of functions as "no_egpr" or
> whatever.

__attribute__((__target__("no-apxf")), yeah.  Perhaps it could be done 
(kernel-wide) for all noinstr functions, but I think we agree that it's 
not a great idea overall.

> At that point, my vote would be to use regs[] to track R16-R31 for KVM's purposes.
> IIUC, we could largely ignore XSAVE state at runtime and just ensure R16-R31 are
> copied to/from userspace as needed, same as we do for PKRU.

If it can be done efficiently when APX is not available, I suppose 
that's fine.  The KVM code is nicer for sure.

But until the kernel starts using APX, I would do the save/restore near 
kvm_load_xfeatures(), because __vmx_vcpu_run()/__svm_vcpu_run() would 
have to check whether xcr0.apx is set or not.  This is much simpler:

	// runs with host xcr0, so it can assume it includes APX
	// alternatively it could be a static_call(), to only invoke
	// the function if at least one guest enables APX in xcr0
	if (static_cpu_has(X86_FEATURE_APX))
		kvm_apx_save(vcpu);
	kvm_load_xfeatures(vcpu, true);
	...
	kvm_load_xfeatures(vcpu, false);
	// same as above
	if (static_cpu_has(X86_FEATURE_APX))
		kvm_apx_restore(vcpu);

Writing kvm_apx_save() and kvm_apx_restore() already now in a .S file is 
fine but I don't care too much.

If the kernel starts using APX we would have to do the xcr0 changes in 
two steps, one in kvm_load_xfeatures() and one (finalizing the APX bit) 
right around the assembly code for world switching.  That xcr0 update 
would have to be in kvm_apx_save/restore, which would have to 1) be 
rewritten in assembly 2) be called from within 
__vmx_vcpu_run/__svm_vcpu_run.

But it would be premature to do now the two-step save/restore of xcr0.

> If R16-R31 aren't generally available for C code, then how exactly is APX going
> to be used?

Only in userspace, at least for now.

Paolo

> Understanding the usage rules for R16-R31 seems fundamental to figuring what to
> do in KVM...