stall/hang on 4.15 kernel with some Xeon skylake CPUs and extended APIC

[Rick Warner]

Hi All,

I've discovered that some new Supermicro skylake systems will hang/stall
while booting the 4.15 kernel when extended APIC (x2apic) is enabled in
the BIOS. The issue happens on specific CPUs only and follows the CPUs.

We had (4) quad socket systems with Xeon 6134 CPUs; 2 out of 4 were
exhibiting this behavior. We replaced 2 CPUs at that time and the
behavior was eliminated. Those systems were then shipped to our customer
(we are an HPC system integrator).

Now, we have 5 single socket systems with 5122 CPUs. 2 out of the 5 are
hanging. If we swap the CPUs from the hanging systems with working
systems, the behavior follows the CPU.

I've done a git bisect between 4.14 and 4.15 and found this commit is
triggering the issue:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?id=023a611748fd58d46c8aa049cf4f22ebada983f5

Some of the commits right before it also seemed to trigger this warning:
[ÂÂÂ 5.062563] Debug warning: early ioremap leak of 1 areas detected.
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ please boot with early_ioremap_debug and report the dmesg.

I have a dmesg log of 1 commit prior to the referenced link with
early_ioremap_debug enabled if it is desired.

The latest git still has the issue.

I've attached a dmesg log captured via serial console from a system
exhibiting this problem. Here is an excerpt from it where the problems
start:

ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
INFO: rcu_sched self-detected stall on CPU
ÂÂÂÂÂÂÂ 34-....: (14997 ticks this GP) idle=b3e/140000000000001/0
softirq=18/18 fqs=7497
INFO: rcu_sched detected stalls on CPUs/tasks:

ÂÂÂÂÂÂÂ 34-....: (14997 ticks this GP) idle=b3e/140000000000001/0
softirq=18/18 fqs=7498
Â(t=15002 jiffies g=-294 c=-295 q=391)
ÂÂÂÂÂÂÂ (detected by 0, t=15002 jiffies, g=-294, c=-295, q=391)
NMI backtrace for cpu 34
CPU: 34 PID: 1 Comm: swapper/0 Not tainted 4.15.7-gentoo-r1-netuno-x86_64 #4
Hardware name: Supermicro SYS-2049U-TR4/X11QPH+, BIOS 2.0c 02/23/2018
Call Trace:
Â<IRQ>
Âdump_stack+0x5d/0x79
Ânmi_cpu_backtrace+0x94/0xae
Â? irq_force_complete_move+0x6f/0x6f
Ânmi_trigger_cpumask_backtrace+0x56/0xd3
Ârcu_dump_cpu_stacks+0x96/0xc0
Ârcu_check_callbacks+0x285/0x697
Âupdate_process_times+0x28/0x4a
Âtick_handle_periodic+0x20/0x5f
Âsmp_apic_timer_interrupt+0x93/0xf9
Âapic_timer_interrupt+0x7d/0x90
Â</IRQ>
RIP: 0010:smp_call_function_many+0x1f1/0x204
RSP: 0000:ffffc900000f3af0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff11
RAX: 0000000000000001 RBX: ffff880c110a0488 RCX: 0000000000000001
RDX: ffff880c10e64440 RSI: 0000000000000000 RDI: ffff880c110a0488
RBP: ffff880c110a0480 R08: fffffffffffffffe R09: 0000000000000003
R10: 0000000000000000 R11: ffffea00c03c1a60 R12: 0000000000000001
R13: ffff880c110a04b8 R14: 0000000000020440 R15: ffffffff81ed5400
Â? slub_cpu_dead+0xa0/0xa0
Â? slub_cpu_dead+0xa0/0xa0
Â? __mmu_notifier_mm_destroy+0x32/0x32
Âon_each_cpu_mask+0x23/0x53
Â? slub_cpu_dead+0xa0/0xa0
Âon_each_cpu_cond+0x7c/0x8b
Â__kmem_cache_shrink+0x3c/0x237
Â? acpi_ps_delete_parse_tree+0x2d/0x59
Â? set_debug_rodata+0x11/0x11
Â? acpi_os_purge_cache+0xa/0xd
Âacpi_os_purge_cache+0xa/0xd
Âacpi_purge_cached_objects+0x29/0x38
Âacpi_initialize_objects+0x46/0x4f
Â? acpi_sleep_init+0xd6/0xd6
Âacpi_init+0xb6/0x324
Â? scan_for_dmi_ipmi+0x15/0xec
Â? acpi_sleep_init+0xd6/0xd6
Âdo_one_initcall+0x89/0x128
Â? set_debug_rodata+0x11/0x11
Â? set_debug_rodata+0x11/0x11
Âkernel_init_freeable+0x112/0x18e
Â? rest_init+0xaa/0xaa
Âkernel_init+0xa/0xf0
Âret_from_fork+0x35/0x40

The NMI dump info repeats periodically after that but never progresses
further.

If any other information is needed, please let me know. I've reported
this issue to Supermicro already and they believe it is an issue with
the kernel opposed to an issue specific to their systems. I don't have
any other brand Xeon skylake systems with extended APIC support that I
can try this with.

Thanks,
Rick


Richard Warner
Chief Technology Officer
Microway, Inc

Attachment: dmesg-hang-with-extended-APIC-enabled.txt.gz
Description: application/gzip