Re: [PATCH] x86/bhi: avoid hardware mitigation for 'spectre_bhi=vmexit'

[Jon Kohler]

> On Sep 13, 2024, at 1:28 AM, Chao Gao <chao.gao@xxxxxxxxx> wrote:
> 
> !-------------------------------------------------------------------|
>  CAUTION: External Email
> 
> |-------------------------------------------------------------------!
> 
> On Thu, Sep 12, 2024 at 09:24:40AM -0700, Pawan Gupta wrote:
>> On Thu, Sep 12, 2024 at 03:44:38PM +0000, Jon Kohler wrote:
>>>> It is only worth implementing the long sequence in VMEXIT_ONLY mode if it is
>>>> significantly better than toggling the MSR.
>>> 
>>> Thanks for the pointer! I hadn’t seen that second sequence. I’ll do measurements on
>>> three cases and come back with data from an SPR system.
>>> 1. as-is (wrmsr on entry and exit)
>>> 2. Short sequence (as a baseline)
>>> 3. Long sequence
>> 

Pawan,

Thanks for the pointer to the long sequence. I've tested it along with 
Listing 3 (TSX Abort sequence) using KUT tscdeadline_immed test. TSX 
abort sequence performs better unless BHI mitigation is off or 
host/guest spec_ctrl values match, avoiding WRMSR toggling. Having the
values match the DIS_S value is easier said than done across a fleet
that is already using eIBRS heavily.

Test System:
- Intel Xeon Gold 6442Y, microcode 0x2b0005c0
- Linux 6.6.34 + patches, qemu 8.2
- KVM Unit Tests @ latest (17f6f2fd) with tscdeadline_immed + edits:
- Toggle spec ctrl before test in main()
- Use cpu type SapphireRapids-v2

Test string:
TESTNAME=vmexit_tscdeadline_immed TIMEOUT=90s MACHINE= ACCEL= taskset -c 26 ./x86/run x86/vmexit.flat \
-smp 1 -cpu SapphireRapids-v2,+x2apic,+tsc-deadline -append tscdeadline_immed |grep tscdeadline

Test Results:
1. spectre_bhi=on, host spec_ctrl=1025, guest spec_ctrl=1: tscdeadline_immed 3878 (WRMSR toggling)
2. spectre_bhi=on, host spec_ctrl=1025, guest spec_ctrl=1025: tscdeadline_immed 3153 (no WRMSR toggling)
3. spectre_bhi=vmexit, BHB long sequence, host/guest spec_ctrl=1: tscdeadline_immed 3629 (still better than test 1, penalty only on exit)
4. spectre_bhi=vmexit, TSX abort sequence, host/guest spec_ctrl=1: tscdeadline_immed 3294 (best general purpose performance)
5. spectre_bhi=vmexit, TSX abort sequence, host spec_ctrl=1, guest spec_ctrl=1025: tscdeadline_immed 4011 (needs optimization)

In short, there is a significant speedup to be had here.

As for test 5, honest that is somewhat invalid because it would be
dependent on the VMM user space showing BHI_CTRL.

QEMU as an example does not do that, so even with latest qemu and latest
kernel, guests will still use BHB loop even on SPR++ today, and they
could use the TSX loop with this proposed change if the VMM exposes RTM
feature.

I'm happy to post a V2 patch with my TSX changes, or take any other
suggestions here. 

Thanks all,
Jon

>> I wonder if virtual SPEC_CTRL feature introduced in below series can
>> provide speedup, as it can replace the MSR toggling with faster VMCS
>> operations:
> 
> "virtual SPEC_CTRL" won't provide speedup. the wrmsr on entry/exit is still
> need if guest's (effective) value and host's value are different.
> 
> "virtual SPEC_CTRL" just prevents guests from toggling some bits. It doesn't
> switch the MSR between guest value and host value on entry/exit. so, KVM has
> to do the switching with wrmsr/rdmsr instructions. A new feature, "load
> IA32_SPEC_CTRL" VMX control (refer to Chapter 15 in ISE spec[*]), can help but
> it isn't supported on SPR.
> 
> [*]: https://urldefense.proofpoint.com/v2/url?u=https-3A__cdrdv2.intel.com_v1_dl_getContent_671368&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=NGPRGGo37mQiSXgHKm5rCQ&m=c7SFjczyXeO5McE4firUZaiOVuLBVwLXAzKV9WQqMqKCCEwSvVk0V4cko-falQYo&s=-hskrlhrR4iuT2sz0KkGJn7hCSAGIteu3_TGQzPgh8I&e= 
> 
>> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel.org_kvm_20240410143446.797262-2D1-2Dchao.gao-40intel.com_&d=DwIDaQ&c=s883GpUCOChKOHiocYtGcg&r=NGPRGGo37mQiSXgHKm5rCQ&m=c7SFjczyXeO5McE4firUZaiOVuLBVwLXAzKV9WQqMqKCCEwSvVk0V4cko-falQYo&s=rsaEdAN9KEjtAMSN-ke4x4R87FgfxsvCsdwbCFk7VOE&e= 
>> 
>> Adding Chao for their opinion.