Re: [PATCH v2 1/2] x86/cpu: Disable CR pinning during CPU bringup

[Borislav Petkov]

On Thu, Feb 26, 2026 at 09:23:48AM +0000, Nikunj A Dadhania wrote:
> From: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
> 
> == CR Pinning Background ==
> 
> Modern CPU hardening features like SMAP/SMEP are enabled by flipping
> control register (CR) bits. Attackers find these features inconvenient and
> often try to disable them.
> 
> CR-pinning is a kernel hardening feature that detects when
> security-sensitive control bits are flipped off, complains about it, then
> turns them back on. The CR-pinning checks are performed in the CR
> manipulation helpers.
> 
> X86_CR4_FRED controls FRED enabling and is pinned. There is a single,
> system-wide static key that controls CR-pinning behavior. The static key is
> enabled by the boot CPU after it has established its CR configuration.
> 
> The end result is that CR-pinning is not active while initializing the boot
> CPU but it is active while bringing up secondary CPUs.
> 
> == FRED Background ==
> 
> FRED is a new hardware entry/exit feature for the kernel. It is not on by
> default and started out as Intel-only. AMD is just adding support now.
> 
> FRED has MSRs for configuration and is enabled by the pinned X86_CR4_FRED
> bit. It should not be enabled until after MSRs are properly initialized.
> 
> == SEV Background ==
> 
> AMD SEV-ES and SEV-SNP use #VC (Virtualization Communication) exceptions to
> handle operations that require hypervisor assistance. These exceptions
> occur during various operations including MMIO access, CPUID instructions,
> and certain memory accesses.
> 
> Writes to the console can generate #VC.
> 
> == Problem ==
> 
> CR-pinning implicitly enables FRED on secondary CPUs at a different point
> than the boot CPU. This point is *before* the CPU has done an explicit
> cr4_set_bits(X86_CR4_FRED) and before the MSRs are initialized. This means
> that there is a window where no exceptions can be handled.
> 
> For SEV-ES/SNP and TDX guests, any console output during this window
> triggers #VC or #VE exceptions that result in triple faults because the
> exception handlers rely on FRED MSRs that aren't yet configured.
> 
> == Fix ==
> 
> Defer CR-pinning enforcement during secondary CPU bringup. This avoids any
> implicit CR changes during CPU bringup, ensuring that FRED is not enabled
> before it is configured and able to handle a #VC or #VE.
> 
> This also aligns boot and secondary CPU bringup.
> 
> CR-pinning is now enforced only when the CPU is online. cr4_init() is
> called during secondary CPU bringup, while the CPU is still offline, so the
> pinning logic in cr4_init() is redundant. Remove it and add WARN_ON_ONCE()
> to catch any future break of this assumption.
> 
> Note: FRED is not on by default anywhere so this is not likely to be
> causing many problems. The only reason this was noticed was that AMD
> started to enable FRED and was turning it on.
> 
> Fixes: 14619d912b65 ("x86/fred: FRED entry/exit and dispatch code")
> Reported-by: Nikunj A Dadhania <nikunj@xxxxxxx>
> Signed-off-by: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
> Signed-off-by: Nikunj A Dadhania <nikunj@xxxxxxx>
> [ Nikunj: Updated SEV background section wording ]
> Reviewed-by: Sohil Mehta <sohil.mehta@xxxxxxxxx>
> Cc: stable@xxxxxxxxxxxxxxx # 6.9+
> ---
>  arch/x86/kernel/cpu/common.c | 23 +++++++++++++++++++----
>  1 file changed, 19 insertions(+), 4 deletions(-)

My SNP guest stops booting with this right:

...
[    3.134372] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP
[    3.138211] SEV: Status: SEV SEV-ES SEV-SNP 
[    3.142211] pid_max: default: 32768 minimum: 301
[    3.145350] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    3.146222] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    3.150613] VFS: Finished mounting rootfs on nullfs
[    3.154971] Running RCU synchronous self tests
[    3.158212] Running RCU synchronous self tests
[    3.269518] smpboot: CPU0: AMD EPYC-v4 Processor (family: 0x17, model: 0x1, stepping: 0x2)
[    3.270237] SEV: APIC: wakeup_secondary_cpu() replaced with wakeup_cpu_via_vmgexit()
[    3.274786] Performance Events: Fam17h+ core perfctr, AMD PMU driver.
[    3.278228] ... version:                   0
[    3.282211] ... bit width:                 48
[    3.284781] ... generic counters:          6
[    3.286212] ... generic bitmap:            000000000000003f
[    3.290211] ... fixed-purpose counters:    0
[    3.294211] ... fixed-purpose bitmap:      0000000000000000
[    3.297549] ... value mask:                0000ffffffffffff
[    3.298211] ... max period:                00007fffffffffff
[    3.302211] ... global_ctrl mask:          000000000000003f
[    3.306490] signal: max sigframe size: 1776
[    3.310343] rcu: Hierarchical SRCU implementation.
[    3.313285] rcu: 	Max phase no-delay instances is 1000.
[    3.314355] Timer migration: 2 hierarchy levels; 8 children per group; 2 crossnode level
[    3.326860] smp: Bringing up secondary CPUs ...
[    3.329547] smpboot: Parallel CPU startup disabled by the platform
[    3.330627] smpboot: x86: Booting SMP configuration:

<--- HERE.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette