Re: [PATCH] x86/fred: Fix early boot failures on SEV-ES/SNP guests

[Dave Hansen]

> CR pinning can prematurely enable features during secondary CPU bringup
> before their supporting infrastructure is initialized. Specifically, when
> FRED is enabled, cr4_init() sets CR4.FRED via the pinned mask early in
> start_secondary(), long before cpu_init_fred_exceptions() configures the
> required FRED MSRs. This creates a window where exceptions cannot be
> properly handled.

This is a collision of FRED, CR-pinning and SEV. Future me would
appreciate having all that background in one place:

	== CR Pinning Background ==

	Modern CPU hardening features like SMAP/SMEP are enabled by
	flipping control register (CR) bits. Attackers find these
	features inconvenient and often try to disable them.

	CR-pinning is a kernel hardening feature that detects when
	security-sensitive control bits are flipped off, complains about
	it, then turns them back on. The CR-pinning checks are performed
	in the CR manipulation helpers.

	X86_CR4_FRED controls FRED enabling and is pinned. There is a
	single, system-wide static key that controls CR-pinning
	behavior. The static key is enabled by the boot CPU after it has
	established its CR configuration.

	The end result is that CR-pinning is not active while
	initializing the boot CPU but it is active while bringing up
	secondary CPUs.

	== FRED Background ==

	FRED is a new hardware entry/exit feature for the kernel. It is
	not on by default and started out as Intel-only. AMD is just
	adding support now.

	FRED has MSRs for configuration and is enabled by the pinned
	X86_CR4_FRED bit. It should not be enabled until after MSRs are
	properly initialized.

	== SEV Background ==

	Some flavors of AMD SEV have special virtualization exceptions:
	#VC. These exceptions happen in "weird" places like when
	accessing MMIO, running CPUID or even accessing apparently
	normal kernel memory.

	Writes to the console can generate #VC.

	== Problem ==

	CR-pinning implicitly enables FRED on secondary CPUs at a
	different point than the boot CPU. This point is *before* the
	CPU has done an explicit cr4_set_bits(X86_CR4_FRED) and before
	the MSRs are initialized. This means that there is a window
	where no exceptions can be handled.

	For SEV-ES/SNP and TDX guests, any console output during this
	window triggers #VC or #VE exceptions that result in triple
	faults because the exception handlers rely on FRED MSRs that
	aren't yet configured.

	== Fix ==

	Defer CR-pinning enforcement during secondary CPU bringup. This
	avoids any implicit CR changes during CPU bringup, ensuring that
	FRED is not enabled before it is configured and able to handle a
	 #VC.

	This also aligns boot and secondary CPU bringup.

	Note: FRED is not on by default anywhere so this is not likely
	to be causing many problems. The only reason this was noticed
	was that AMD started to enable FRED and was turning it on.

With that, you can add:

Signed-off-by: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>