Re: Fw: xAPIC deprecation plan - concerns from a bootloader/OS developer

[Andrew Cooper]

> Hi all,
>
> I'm the author of Limine, a multiprotocol bootloader that handles SMP
> bring-up across a wide range of x86 hardware. I wanted to raise some
> concerns about Intel's xAPIC deprecation plan and its real-world
> impact on the x86 ecosystem.
>
> The deprecation is apparently motivated by AEPIC Leak
> (CVE-2022-21233), where reads to undefined offsets in the xAPIC MMIO
> page return stale data from the cache hierarchy. This is a serious
> issue for SGX, since the SGX threat model assumes a hostile OS that
> can freely access MMIO. The recommended mitigation is to use x2APIC
> mode, which replaces the MMIO interface with MSR-based access.

I was involved in the coordination of CVE-2022-21233, but am not
affiliated with Intel.  I have opinions on how it was handled.

AIUI, there was a vague longterm deprecation plan, which got actioned in
a hurry to address ÆPIC Leak.


> However, this vulnerability is specific to Intel's microarchitectural
> implementation. It is not an architectural flaw in the xAPIC
> specification itself. AMD implements xAPIC MMIO correctly and has no
> equivalent vulnerability, no deprecation plan, and no intention of
> removing xAPIC support. The bug was also fixed in silicon starting
> with Sapphire Rapids. So the deprecation is not addressing a
> fundamental design problem - it is using an Intel-specific
> implementation bug as justification for an architecture-wide
> compatibility break.
>
> One could have argued that the xAPIC deprecation was a stepping stone
> toward X86S, Intel's proposal to strip legacy modes from the
> architecture entirely, but X86S was killed in December 2024 after the
> formation of the x86 Ecosystem Advisory Group, which now requires
> Intel and AMD to collaborate on ISA changes. With X86S off the table,
> the xAPIC deprecation no longer serves any broader architectural
> simplification goal - it's just a standalone compatibility break with
> no clear endgame beyond itself.

I can't speak to x86-S, but there absolutely are simplifications to be
gained by removing the circuitry for xAPIC mode.

The ability to move the xAPIC MMIO window has caused no end of
vulnerabilities over the decades, perhaps most famously Sinkhole
(disclosed 2015, no CVE, and no public mention from vendors who fixed it
quietly in microcode.)

>From an OS perspective, the fact that xAPIC has ICR split in two leads
to a whole bunch of bugs, including the need to disable interrupts in
order to sent an IPI safely, and why it's not safe to send IPIs from NMI
context.  These are rare-to-hit races causing IPIs to target the wrong
CPUs, which can stay hidden for years.  There's a very good reason why
x2APIC reworked ICR to be in a single MSR rather than being split like
every other >32bit register.

>From a virt perspective, making OSes which move the MMIO window function
correctly is sufficiently hard that giving up is the easier option,
relying on the fact that nothing tends to move the window in practice.

> The practical consequences are already causing problems.
>
> On Phase 2a platforms (Arrow Lake, Lunar Lake), motherboard vendors
> have introduced BIOS toggles to "disable x2APIC" for compatibility
> with older operating systems. But because of the one-way lock
> mechanism (IA32_XAPIC_DISABLE_STATUS), these toggles don't just boot
> the system in xAPIC mode with the option to switch to x2APIC later -
> they lock the system out of x2APIC entirely. The OS cannot enable
> x2APIC at runtime because the BIOS hides the capability (likely via
> CPUID masking or DMAR opt-out flags). So instead of a graceful "xAPIC
> by default, x2APIC available," users get a binary choice between
> "x2APIC locked on forever" and "x2APIC hidden forever".

You say likely via masking, but which is it precisely?  This question
needs bringing up with Intel.  "Locked into xAPIC mode" doesn't seem
like a sensible capability, but it will probably depend on how MCHECK
works on these platforms.

> I'd be interested to hear whether there has been any discussion within
> the x86 EAG or the kernel community about pushing back on this, or
> whether this is considered a settled matter. From where I sit, the
> ecosystem is paying a real cost for an Intel-specific bug that has
> already been fixed.

No amount of pushing back is going to get xAPIC back into Intel
hardware.  As a consumer, voting with your wallet is the only real option.

Sorry.

Going off on a slight tangent.

What is nasty about this is that "locked into $THIS_MODE_ONLY" would be
great for virt, where MMIO vs MSR makes a substantial difference to the
complexity and performance of the implementation.  There should have
been a lock bit put into MSR_APIC_BASE itself, and a CPUID bit
enumerating this capability.

But instead we got a bit in an Intel-only MSR, enumerated by a different
bit in a different Intel-only MSR, enumerated by a CPUID bit, and I kid
you not the justification I was given for not using a lock bit in
MSR_APIC_BASE was "that would be too invasive to the qualification tests".

So instead of architecting this nicely, we've got more Intel branded
duct tape which can't stick the other useful cases.

~Andrew