Re: Fw: xAPIC deprecation plan - concerns from a bootloader/OS developer

[mintsuki]

> > Hi all,
> >
> > I'm the author of Limine, a multiprotocol bootloader that handles SMP
> > bring-up across a wide range of x86 hardware. I wanted to raise some
> > concerns about Intel's xAPIC deprecation plan and its real-world
> > impact on the x86 ecosystem.
> >
> > The deprecation is apparently motivated by AEPIC Leak
> > (CVE-2022-21233), where reads to undefined offsets in the xAPIC MMIO
> > page return stale data from the cache hierarchy. This is a serious
> > issue for SGX, since the SGX threat model assumes a hostile OS that
> > can freely access MMIO. The recommended mitigation is to use x2APIC
> > mode, which replaces the MMIO interface with MSR-based access.
> 
> I was involved in the coordination of CVE-2022-21233, but am not
> affiliated with Intel.  I have opinions on how it was handled.
> 
> AIUI, there was a vague longterm deprecation plan, which got actioned in
> a hurry to address ÆPIC Leak.
> 
> 
> > However, this vulnerability is specific to Intel's microarchitectural
> > implementation. It is not an architectural flaw in the xAPIC
> > specification itself. AMD implements xAPIC MMIO correctly and has no
> > equivalent vulnerability, no deprecation plan, and no intention of
> > removing xAPIC support. The bug was also fixed in silicon starting
> > with Sapphire Rapids. So the deprecation is not addressing a
> > fundamental design problem - it is using an Intel-specific
> > implementation bug as justification for an architecture-wide
> > compatibility break.
> >
> > One could have argued that the xAPIC deprecation was a stepping stone
> > toward X86S, Intel's proposal to strip legacy modes from the
> > architecture entirely, but X86S was killed in December 2024 after the
> > formation of the x86 Ecosystem Advisory Group, which now requires
> > Intel and AMD to collaborate on ISA changes. With X86S off the table,
> > the xAPIC deprecation no longer serves any broader architectural
> > simplification goal - it's just a standalone compatibility break with
> > no clear endgame beyond itself.
> 
> I can't speak to x86-S, but there absolutely are simplifications to be
> gained by removing the circuitry for xAPIC mode.
> 
> The ability to move the xAPIC MMIO window has caused no end of
> vulnerabilities over the decades, perhaps most famously Sinkhole
> (disclosed 2015, no CVE, and no public mention from vendors who fixed it
> quietly in microcode.)
> 
> From an OS perspective, the fact that xAPIC has ICR split in two leads
> to a whole bunch of bugs, including the need to disable interrupts in
> order to sent an IPI safely, and why it's not safe to send IPIs from NMI
> context.  These are rare-to-hit races causing IPIs to target the wrong
> CPUs, which can stay hidden for years.  There's a very good reason why
> x2APIC reworked ICR to be in a single MSR rather than being split like
> every other >32bit register.
> 
> From a virt perspective, making OSes which move the MMIO window function
> correctly is sufficiently hard that giving up is the easier option,
> relying on the fact that nothing tends to move the window in practice.
> 
> > The practical consequences are already causing problems.
> >
> > On Phase 2a platforms (Arrow Lake, Lunar Lake), motherboard vendors
> > have introduced BIOS toggles to "disable x2APIC" for compatibility
> > with older operating systems. But because of the one-way lock
> > mechanism (IA32_XAPIC_DISABLE_STATUS), these toggles don't just boot
> > the system in xAPIC mode with the option to switch to x2APIC later -
> > they lock the system out of x2APIC entirely. The OS cannot enable
> > x2APIC at runtime because the BIOS hides the capability (likely via
> > CPUID masking or DMAR opt-out flags). So instead of a graceful "xAPIC
> > by default, x2APIC available," users get a binary choice between
> > "x2APIC locked on forever" and "x2APIC hidden forever".
> 
> You say likely via masking, but which is it precisely?  This question
> needs bringing up with Intel.  "Locked into xAPIC mode" doesn't seem
> like a sensible capability, but it will probably depend on how MCHECK
> works on these platforms.

This is honestly less of a concern as it only affects these platforms and
it doesn't really address my broader concerns about the xAPIC deprecation
and removal overall. It would be nice to have it sorted out, but it's not
a big priority of mine.

> 
> > I'd be interested to hear whether there has been any discussion within
> > the x86 EAG or the kernel community about pushing back on this, or
> > whether this is considered a settled matter. From where I sit, the
> > ecosystem is paying a real cost for an Intel-specific bug that has
> > already been fixed.
> 
> No amount of pushing back is going to get xAPIC back into Intel
> hardware.  As a consumer, voting with your wallet is the only real option.
> 
> Sorry.

Funny, I was told this exact same thing about X86S by multiple people, and
then X86S ended up getting shelved.

I need to test on multiple hardware from multiple vendors so I am personally
in no position to "vote with my wallet", as much as I'd personally like to.

I think that breaking compatibility with all OSes/bootloaders that rely on
the xAPIC (funnily enough, not those capable of relying solely on the legacy
PIC, at least in theory, we'll see) is a pretty insane compatibility break.

I love the x2APIC, I agree with you about pretty much everything you've said
in that regard. My problem isn't with the x2APIC, but rather with the sudden
and surgical (only removing the xAPIC) compatibility break for dubious gains
in the aspect of security.

I appreciate your input, though!

> 
> Going off on a slight tangent.
> 
> What is nasty about this is that "locked into $THIS_MODE_ONLY" would be
> great for virt, where MMIO vs MSR makes a substantial difference to the
> complexity and performance of the implementation.  There should have
> been a lock bit put into MSR_APIC_BASE itself, and a CPUID bit
> enumerating this capability.
> 
> But instead we got a bit in an Intel-only MSR, enumerated by a different
> bit in a different Intel-only MSR, enumerated by a CPUID bit, and I kid
> you not the justification I was given for not using a lock bit in
> MSR_APIC_BASE was "that would be too invasive to the qualification tests".
> 
> So instead of architecting this nicely, we've got more Intel branded
> duct tape which can't stick the other useful cases.
> 
> ~Andrew
>