Re: [PATCH v14 10/13] KVM: x86: Enable CET virtualization for VMX and advertise CET to userspace

[Yang Weijiang]

On Fri, Jan 29, 2021 at 03:38:52PM +0100, Paolo Bonzini wrote:
> On 29/01/21 13:17, Yang Weijiang wrote:
> > > > It's specific to VM case, during VM reboot, memory mode reset but VM_ENTRY_LOAD_CET_STATE
> > > > is still set, and VMCS contains stale GUEST_SSP, this hits vm-entry failure
> > > > documented in 10.7 VM Entry at:
> > > > https://software.intel.com/sites/default/files/managed/4d/2a/control-flow-enforcement-technology-preview.pdf
> > > > Since CR4.CET is also reset during VM reboot, to take the change to clear the stale data.
> > > > Maybe I need to find a better place to do the things.
> > > Then you must use a field of struct vmx_vcpu instead of the VMCS to hold
> > > GUEST_SSP (while GUEST_S_CET and GUEST_INTR_SSP_TABLE should not be an
> > > issue).
> > > 
> > Sorry, I don't get your point, can I just clear the GUEST_SSP field in this case?
> > Anyway save/restore GUEST_SSP via VMCS is an efficient way.
> 
> You cannot clear it, because it is preserved when CR4.CET is modified.
> 
> However, I checked the latest SDM and the GUEST_SSP rules are changed to
> just this:
> 
> SSP. The following checks are performed if the “load CET state” VM-entry
> control is 1
> — Bits 1:0 must be 0.
> — If the processor supports the Intel 64 architecture, bits 63:N must be
> identical, where N is the CPU’s maximum linear-address width. (This check
> does not apply if the processor supports 64 linear-address bits.) The guest
> SSP value is not required to be canonical; the value of bit N-1 may differ
> from that of bit N.
> 
> In particular it doesn't mention the "IA-32e mode guest" VM-entry control or
> the CS.L bit anymore, so it should not be necessary anymore to even reset
> SSP to 0, and you can keep GUEST_SSP in the VMCS.
>
There could be some gaps between the two specs, I tested VM reboot with these patches,
if I don't clear GUEST_SSP field while CR4.CET is changed, then it always encounters the
vm-entry failure issue, the VMCS dump is like below:

[341485.265277] *** Guest State ***
[341485.265280] CR0: actual=0x0000000000000030,
shadow=0x0000000060000010, gh_mask=fffffffffffffff7
[341485.265281] CR4: actual=0x0000000000002040,
shadow=0x0000000000000000, gh_mask=fffffffffffef871
[341485.265282] CR3 = 0x0000000000000000
[341485.265283] RSP = 0x0000000000000000  RIP = 0x000000000000fff0
[341485.265283] RFLAGS=0x00000002         DR7 = 0x0000000000000400
[341485.265284] Sysenter RSP=0000000000000000
CS:RIP=0000:0000000000000000
[341485.265285] CS:   sel=0xf000, attr=0x0009b, limit=0x0000ffff,
base=0x00000000ffff0000
[341485.265286] DS:   sel=0x0000, attr=0x00093, limit=0x0000ffff,
base=0x0000000000000000
[341485.265287] SS:   sel=0x0000, attr=0x00093, limit=0x0000ffff,
base=0x0000000000000000
[341485.265288] ES:   sel=0x0000, attr=0x00093, limit=0x0000ffff,
base=0x0000000000000000
[341485.265288] FS:   sel=0x0000, attr=0x00093, limit=0x0000ffff,
base=0x0000000000000000
[341485.265289] GS:   sel=0x0000, attr=0x00093, limit=0x0000ffff,
base=0x0000000000000000
[341485.265289] GDTR:                           limit=0x0000ffff,
base=0x0000000000000000
[341485.265290] LDTR: sel=0x0000, attr=0x00082, limit=0x0000ffff,
base=0x0000000000000000
[341485.265291] IDTR:                           limit=0x0000ffff,
base=0x0000000000000000
[341485.265291] TR:   sel=0x0000, attr=0x0008b, limit=0x0000ffff,
base=0x0000000000000000
[341485.265292] EFER =     0x0000000000000000  PAT = 0x0007040600070406
[341485.265292] DebugCtl = 0x0000000000000000  DebugExceptions =
0x0000000000000000
[341485.265293] Interruptibility = 00000000  ActivityState = 00000000
[341485.265294] InterruptStatus = 0000
[341485.265294] S_CET = 0x0000000000000000
[341485.265295] SSP = 0x00007fc1727fdfd0
[341485.265295] SSP TABLE = 0x0000000000000000
[341485.265296] *** Host State ***
[341485.265296] RIP = 0xffffffffc1235900  RSP = 0xffffaed7c150bd68
[341485.265297] CS=0010 SS=0018 DS=0000 ES=0000 FS=0000 GS=0000 TR=0040
[341485.265298] FSBase=00007f6851d36700 GSBase=ffff9de2e0980000
TRBase=fffffe0000141000
[341485.265298] GDTBase=fffffe000013f000 IDTBase=fffffe0000000000
[341485.265299] CR0=0000000080050033 CR3=00000001135ae001
CR4=0000000000f72ee0
[341485.265300] Sysenter RSP=fffffe0000141000
CS:RIP=0010:ffffffff8dc015e0
[341485.265301] EFER = 0x0000000000000d01  PAT = 0x0407050600070106
[341485.265301] *** Control State ***
[341485.265302] PinBased=000000ff CPUBased=b5a06dfa
SecondaryExec=021237eb
[341485.265303] EntryControls=0010d1ff ExitControls=102befff
[341485.265303] ExceptionBitmap=00060042 PFECmask=00000000
PFECmatch=00000000
[341485.265304] VMEntry: intr_info=00000000 errcode=00000000
ilen=00000000
[341485.265305] VMExit: intr_info=00000000 errcode=00000000
ilen=00000002
[341485.265306]         reason=80000021 qualification=0000000000000000
[341485.265306] IDTVectoring: info=00000000 errcode=00000000
[341485.265307] TSC Offset = 0xfffd10acb111a2a0
[341485.265307] TSC Multiplier = 0x0001000000000000
[341485.265308] SVI|RVI = 00|00 TPR Threshold = 0x00
[341485.265309] APIC-access addr = 0x0000000499d63000 virt-APIC addr =
0x000000011ade6000
[341485.265310] PostedIntrVec = 0xf2
[341485.265310] EPT pointer = 0x000000011fffc05e
[341485.265310] PLE Gap=00000080 Window=00010000
[341485.265311] Virtual processor ID = 0x0001
[341485.265312] S_CET = 0x0000000000000000
[341485.265312] SSP = 0x0000000000000000
[341485.265312] SSP TABLE = 0x0000000000000000

The alternative way is to clear it in exit_lmode(), it also works.  

> Paolo