Re: [PATCH 1/1] x86: SMP broken on Xen PV DomU since 6.9

[Niels Dettenbach]

Virtual machines under Xen Hypervisor (DomU) running in Xen PV mode use a 
special, nonstandard synthetized CPU topology which "just works" under 
kernels 6.9.x while newer kernels wrongly assuming a "crash kernel" and 
disable SMP (reducing to one CPU core) because the newer topology 
implementation produces a wrong error "[Firmware Bug]: APIC enumeration 
order not specification compliant" after new topology checks which are 
improper for Xen PV platform. As a result, the kernel disables SMP and 
activates just one CPU core within the PV DomU "VM" (DomU in PV mode).

The patch disables the regarding checks if it is running in Xen PV 
mode (only) and bring back SMP / all CPUs as in the past to such DomU 
VMs. The Xen subsystem takes care of the proper interaction between "guest" 
(DomU) and the "host" (Dom0).

Signed-off-by: Niels Dettenbach <nd@xxxxxxxxxxxx>

---


The current behaviour leads all of our production Xen Host platforms 
(amd64 - HPE proliant) unusable after updating to newer linux kernels 
(with just one CPU available/activated per VM) while older kernels and
other OS (current NetBSD PV DomU) still work fully (and stable since many 
years on the platform). 

Xen PV mode is still provided by current Xen and widely used - even 
if less wide as the newer Xen PVH mode today. So a solution probably 
will be required and the other exemptions in topology.c seems to support that 
as well.

So we assume that bug affects stable@xxxxxxxxxxxxxxx as well.


dmesg from affected kernel on affected DomU PV:

-- snip --
[    0.640364] CPU topo: Enumerated BSP APIC 0 is not marked in APICBASE MSR
[    0.640367] CPU topo: Assuming crash kernel. Limiting to one CPU to 
prevent machine INIT
[    0.640368] CPU topo: [Firmware Bug]: APIC enumeration order not 
specification compliant
[    0.640376] CPU topo: Max. logical packages:   1
[    0.640378] CPU topo: Max. logical dies:       1
[    0.640379] CPU topo: Max. dies per package:   1
[    0.640386] CPU topo: Max. threads per core:   1
[    0.640388] CPU topo: Num. cores per package:     1
[    0.640389] CPU topo: Num. threads per package:   1
[    0.640390] CPU topo: Allowing 1 present CPUs plus 0 hotplug CPUs
[    0.640402] Cannot find an available gap in the 32-bit address range
-- snap --


after patch applied:
-- snip --
[    0.369439] CPU topo: Max. logical packages:   1
[    0.369441] CPU topo: Max. logical dies:       1
[    0.369442] CPU topo: Max. dies per package:   1
[    0.369450] CPU topo: Max. threads per core:   2
[    0.369452] CPU topo: Num. cores per package:     3
[    0.369453] CPU topo: Num. threads per package:   6
[    0.369453] CPU topo: Allowing 6 present CPUs plus 0 hotplug CPUs
-- snap --

We tested the patch intensely under productive / high load since 2 weeks now 
with no issues (no crashes emulated).


references:

arch/x86/kernel/cpu/topology.c
[line 448]
-- snip --
        /*
         * XEN PV is special as it does not advertise the local APIC
         * properly, but provides a fake topology for it so that the
         * infrastructure works. So don't apply the restrictions vs. APIC
         * here.
         */
--snap --


Am Dienstag, 1. Oktober 2024, 14:28:58  schrieben Sie:
> Please Cc LKML on such things as docuemnted...
> 
> On Mon, Sep 30 2024 at 09:43, Niels Dettenbach wrote:
> > Virtual machines under Xen Hypervisor (DomU) running in Xen PV mode use a
> > special, nonstandard synthetized CPU topology which "just works" under
> > kernels 6.9.x while newer kernels wrongly assuming a "crash kernel" and
> > disable SMP (reducing to one CPU core) because the newer topology
> > implementation produces a wrong error "[Firmware Bug]: APIC enumeration
> > order not specification compliant" after new topology checks which are
> > improper for Xen PV platform.
> 
> Why are they incorrect for XENPV? If the hypervisor exposes a topology
> via CPUID then that topology wants to be correct. Everything else is a
> firmware bug. And no, we are not papering over that.
> 
> > As a result, the kernel disables SMT and activates just one CPU core
> > within the VM (DomU).
> 
> I'm not seing how that happens due to the firmware bug issue. That's a
> different problem and has nothing to do with
> topology_register[_boot]_apic().
> 
> > The patch disables the regarding checks if it is running in Xen PV
> > mode (only) and bring back SMP / all CPUs as in the past to such DomU
> > VMs.
> 
> So what enumerates the APICs on your systems? Is the topology exposed
> via ACPI? If not, then this can't happen at all.
> 
> And please report this with LKML cc'ed with the above questions
> answered.

Hi Thomas,

my patch just extends existing exceptions for xen in topology.c

i took the exact inline explanation of my patch from existing (so i assume 
approved) code in topology.c - please have a look (line 449 of arch/x86/
kernel/cpu/topology.c).

xen is "part" of the linux kernel and in PV mode DomU (guest) the xen Dom0 
(„host“) apic topology is "emulated" by Xen itself in a „non-standard“ but 
working structure while Xen in PV DomU takes care of the frontend side. The 
patch only affects this very special constellation / setup.


In production environments different kernel / xen versions typically are used 
- means: even if the Xen project will „fix“ / or adapt to that new behaviour 
of linux kernel, it still breaks compatibility on affected existing Xen (PV) 
productive installations for at least several years.


The >6.9 kernel runs SMP fine if i boot it on the same bare metal of that 
different HP proliant machines (means instead of booting it as Xen PV guest 
onto a Linux xen Dom0 on that hardware) so the hardwares firmware could not 
be the issue here, while it is affected in Xen DomU VM guests (in PV mode) 
only on different hardware with different (but older) Xen versions / binaries 
(in Xen PV mode only). So from my prospective and as far as i read  the whole 
topology.c (i'm not a skilled kernel developer) which helds several such 
exemptions from different tests/checks / fnctions this is a (known) Xen 
specialty.

I know that Xen PV mode gets more succeeded by the newer xen PVHM mode these 
days, but PV mode is officially supported by Xen as linux kernel and still 
used in productive scenarios.

see i.e.: 
https://xen-orchestra.com/blog/xen-virtualization-modes/
https://wiki.xenproject.org/wiki/Understanding_the_Virtualization_Spectrum



many thanks for your time,


niels.




--- linux/arch/x86/kernel/cpu/topology.c.orig   2024-09-11 09:53:16.194095250 
+0200
+++ linux/arch/x86/kernel/cpu/topology.c        2024-09-30 09:39:11.041326786 
+0200
@@ -131,6 +131,18 @@ static __init bool check_for_real_bsp(u3
        bool is_bsp = false, has_apic_base = boot_cpu_data.x86 >= 6;
        u64 msr;

+
+        /*
+         * XEN PV is special as it does not advertise the local APIC
+         * properly, but provides a fake topology for it so that the
+         * infrastructure works. So don't apply the restrictions vs. APIC
+         * here.
+         */
+       if (xen_pv_domain()) {
+               topo_info.real_bsp_apic_id = topo_info.boot_cpu_apic_id;
+               return false;
+       }
+
        /*
         * There is no real good way to detect whether this a kdump()
         * kernel, but except on the Voyager SMP monstrosity which is not