Re: [PATCH v3 0/9] Parallel CPU bringup for x86_64
From: Paul Menzel
Date: Wed Dec 29 2021 - 08:18:36 EST
Dear David,
Am 28.12.21 um 15:18 schrieb David Woodhouse:
On Tue, 2021-12-28 at 12:34 +0100, Paul Menzel wrote:
Same on the ASUS F2A85-M PRO with AMD A6-6400K. Without serial console,
the messages below are printed below to the monitor after nine seconds.
[ 1.078879] smp: Bringing up secondary CPUs ...
[ 1.080950] x86: Booting SMP configuration:
Please find the serial log attached.
Thanks for testing. That looks like the same triple-fault on bringup
that we have been seeing, and that I reproduced without my patches
using kexec all the way back to a 5.0 kernel.
Out of interest, are you also able to reproduce it with kexec and
without the parallel bringup?
No, I am not able to reproduce that with Debian’s
*linux-image-5.15.0-2-686*, and kexec. With this board,
`module_blacklist=radeon` is needed, as the driver *radeon* is not able
to deal with kexec – and amdgpu neither [1].
```
[ 3.349911] [drm] Found VCE firmware/feedback version 50.0.1 / 17!
[ 3.365259] clocksource: Switched to clocksource tsc
[ 3.365284] [drm] GART: num cpu pages 262144, num gpu pages 262144
[ 3.405159] random: fast init done
[ 3.420492] [drm] PCIE GART of 1024M enabled (table at
0x00000000001D6000).
[ 3.427634] radeon 0000:00:01.0: WB enabled
[ 3.431828] radeon 0000:00:01.0: fence driver on ring 0 use gpu addr
0x0000000020000c00
[ 3.440100] radeon 0000:00:01.0: fence driver on ring 5 use gpu addr
0x0000000000075a18
[ 3.458182] radeon 0000:00:01.0: failed VCE resume (-22).
[ 3.463591] radeon 0000:00:01.0: fence driver on ring 1 use gpu addr
0x0000000020000c04
[ 3.471615] radeon 0000:00:01.0: fence driver on ring 2 use gpu addr
0x0000000020000c08
[ 3.479636] radeon 0000:00:01.0: fence driver on ring 3 use gpu addr
0x0000000020000c0c
[ 3.487650] radeon 0000:00:01.0: fence driver on ring 4 use gpu addr
0x0000000020000c10
[ 3.495990] radeon 0000:00:01.0: radeon: MSI limited to 32-bit
[ 3.502008] radeon 0000:00:01.0: radeon: using MSI.
[ 3.506906] ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 3.506918] [drm] radeon: irq initialized.
[
```
And with that patch I sent Tom in
https://lore.kernel.org/lkml/721484e0fa719e99f9b8f13e67de05033dd7cc86.camel@xxxxxxxxxxxxx/
to expand the bitlock exclusion and stop the bringup being truly in
parallel at all?
No, this does not help, and the Linux kernel resets at the same spot.
```
[ 1.036036] smpboot: smpboot: After
x86_init.timers.setup_percpu_clockev()
[ 1.037031] smpboot: smp_get_logical_apicid()
[ 1.038031] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD
Graphics (family: 0x15, model: 0x13, stepping: 0x1)
[ 1.039366] Performance Events: Fam15h core perfctr, AMD PMU driver.
[ 1.040033] ... version: 0
[ 1.041031] ... bit width: 48
[ 1.042031] ... generic registers: 6
[ 1.043033] ... value mask: 0000ffffffffffff
[ 1.044031] ... max period: 00007fffffffffff
[ 1.045031] ... fixed-purpose events: 0
[ 1.046031] ... event mask: 000000000000003f
[ 1.048065] rcu: Hierarchical SRCU implementation.
[ 1.050642] NMI watchdog: Enabled. Permanently consumes one hw-PMU
counter.
[ 1.051133] smp: Bringing up secondary CPUs ...
[ 1.053202] x86: Booting SMP configuration:
```
Or the one in
https://lore.kernel.org/lkml/d4cde50b4aab24612823714dfcbe69bc4bb63b60.camel@xxxxxxxxxxxxx
which makes it do nothing except prepare all the CPUs before bringing
them up one at a time?
I applied it on top the other one, and it made no difference either.
My current theory (not that I've spent that much time thinking about it
in the last week) is that there's something about the existing CPU
bringup, possibly a CPU bug or something special about the AMD CPUs,
which is triggered by just making it a little bit *faster*, which is
why bringing them up from kexec (especially in qemu) can cause it too?
Would having the serial console enabled make a difference?
Tom seemed to find that it was in load_TR_desc(), so if you could try
this hack on a machine that doesn't magically wink out of existence on
a triplefault before even flushing its serial output, that would be
much appreciated...
diff --git a/arch/x86/include/asm/desc.h b/arch/x86/include/asm/desc.h
index ab97b22ac04a..cc6590712ff4 100644
--- a/arch/x86/include/asm/desc.h
+++ b/arch/x86/include/asm/desc.h
@@ -8,7 +8,7 @@
#include <asm/fixmap.h>
#include <asm/irq_vectors.h>
#include <asm/cpu_entry_area.h>
-
+#include <asm/io.h>
#include <linux/debug_locks.h>
#include <linux/smp.h>
#include <linux/percpu.h>
@@ -265,11 +265,16 @@ static inline void native_load_tr_desc(void)
* If the current GDT is the read-only fixmap, swap to the original
* writeable version. Swap back at the end.
*/
+ outb('d', 0x3f8);
if (gdt.address == (unsigned long)fixmap_gdt) {
+ outb('e', 0x3f8);
load_direct_gdt(cpu);
restore = 1;
+ outb('f', 0x3f8);
}
+ outb('g', 0x3f8);
asm volatile("ltr %w0"::"q" (GDT_ENTRY_TSS*8));
+ outb('h', 0x3f8);
if (restore)
load_fixmap_gdt(cpu);
}
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 0083464de5e3..5bc8f30c3283 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -1716,7 +1716,9 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c)
enable_sep_cpu();
#endif
mtrr_ap_init();
+outb('A', 0x3f8);
validate_apic_and_package_id(c);
+outb('B', 0x3f8);
x86_spec_ctrl_setup_ap();
update_srbds_msr();
}
@@ -1957,6 +1959,7 @@ static inline void tss_setup_io_bitmap(struct tss_struct *tss)
tss->io_bitmap.mapall[IO_BITMAP_LONGS] = ~0UL;
#endif
}
+#include <asm/realmode.h>
/*
* Setup everything needed to handle exceptions from the IDT, including the IST
@@ -1969,16 +1972,24 @@ void cpu_init_exception_handling(void)
/* paranoid_entry() gets the CPU number from the GDT */
setup_getcpu(cpu);
-
+ outb('\n', 0x3f8);
+ outb('0' + cpu / 100, 0x3f8);
+ outb('0' + (cpu % 100) / 10, 0x3f8);
+ outb('0' + (cpu % 10), 0x3f8);
+
/* IST vectors need TSS to be set up. */
tss_setup_ist(tss);
+ outb('a', 0x3f8);
tss_setup_io_bitmap(tss);
set_tss_desc(cpu, &get_cpu_entry_area(cpu)->tss.x86_tss);
-
+ outb('b', 0x3f8);
load_TR_desc();
+ outb('c', 0x3f8);
/* Finally load the IDT */
load_current_idt();
+ outb('z', 0x3f8);
+
}
/*
Unfortunately, no more messages were printed on the serial console.
Kind regards,
Paul
[1]: https://gitlab.freedesktop.org/drm/amd/-/issues/1597