Re: linux-next regression: SNP Guest boot hangs with certain cpu/mem config combination
From: Kirill A. Shutemov
Date: Thu Mar 27 2025 - 10:26:46 EST
On Thu, Mar 27, 2025 at 02:58:12PM +0200, Kirill A. Shutemov wrote:
> On Wed, Mar 26, 2025 at 05:30:35PM -0500, Tom Lendacky wrote:
> > On 3/25/25 08:33, Kirill A. Shutemov wrote:
> > > On Tue, Mar 25, 2025 at 02:40:00PM +0530, Aithal, Srikanth wrote:
> > >> Hello,
> > >>
> > >>
> > >> Starting linux-next build next-20250312, including recent build 20250324, we
> > >> are seeing an issue where the SNP guest boot hangs at the "boot smp config"
> > >> step:
> > >>
> > >>
> > >> [ 2.294722] smp: Bringing up secondary CPUs ...
> > >> [ 2.295211] smpboot: Parallel CPU startup disabled by the platform
> > >> [ 2.309687] smpboot: x86: Booting SMP configuration:
> > >> [ 2.310214] .... node #0, CPUs: #1 #2 #3 #4 #5 #6
> > >> #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21
> > >> #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36
> > >> #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51
> > >> #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66
> > >> #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 #80 #81
> > >> #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 #96
> > >> #97 #98 #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111
> > >> #112 #113 #114 #115 #116 #117 #118 #119 #120 #121 #122 #123 #124 #125 #126
> > >> #127 #128 #129 #130 #131 #132 #133 #134 #135 #136 #137 #138 #139 #140 #141
> > >> #142 #143 #144 #145 #146 #147 #148 #149 #150 #151 #152 #153 #154 #155 #156
> > >> #157 #158 #159 #160 #161 #162 #163 #164 #165 #166 #167 #168 #169 #170 #171
> > >> #172 #173 #174 #175 #176 #177 #178 #179 #180 #181 #182 #183 #184 #185 #186
> > >> #187 #188 #189 #190 #191 #192 #193 #194 #195 #196 #197 #198
> > >> --> The guest hangs forever at this point.
> > >>
> > >>
> > >> I have observed that certain vCPU and memory combinations work, while others
> > >> do not. The VM configuration I am using does not have any NUMA nodes.
> > >>
> > >> vcpus Mem SNP guest boot
> > >> <=240 19456M Boots fine
> > >>> =241,<255 19456M Hangs
> > >> 1-255 2048M Boots fine
> > >> 1-255 4096M Boots fine
> > >>> 71 8192M Hangs
> > >>> 41 6144M Hangs
> > >>
> > >> When I bisected this issue, it pointed to the following commit :
> > >>
> > >>
> > >> *commit 800f1059c99e2b39899bdc67a7593a7bea6375d8*
> > >> Author: Kirill A. Shutemov <kirill.shutemov@xxxxxxxxxxxxxxx>
> > >> Date: Mon Mar 10 10:28:55 2025 +0200
> > >>
> > >> mm/page_alloc: fix memory accept before watermarks gets initialized
> > >
> > > Hm. It is puzzling for me. I don't see how this commit can cause the hang.
> > >
> > > Could you track down where hang happens?
> >
> > Let me say that the guest config is key for this. Using that config, I
> > think you might be able to repro this on TDX. The config does turn off TDX
> > support, so I'm hoping that turning it on doesn't change anything.
> >
> > I've been able to track it down slightly... It is happening during the CPU
> > bringup trace points and it eventually gets to line 2273 in
> > rb_allocate_cpu_buffer() and never comes back from an alloc_pages_node()
> > call. That's as far as I've gotten so far. I'm not a mm expert so not sure
> > if I'll be able to progress much further.
>
> Urgh... It is deadlock on cpu_hotplug_lock :/
>
> _cpu_up() takes the lock on write and starts CPU bring up under it.
> If during CPU bringup we accept the last page in the zone, __accept_page()
> calls static_branch_dec() which takes the lock again.
>
> Oopsie.
>
> So the patch itself doesn't introduce a regression, but uncovers
> preexisting deadlock. With the patch we accept more pages during the boot
> and it triggers the deadlock.
>
> Let me think about the fix.
+ Static branch maintainers
The only option I see so far is to drop static branch from this path.
But I am not sure if it the only case were we use static branch from CPU
hotplug callbacks.
Any other ideas?
The deadlock I'm talking about:
============================================
WARNING: possible recursive locking detected
6.14.0-rc5+ #13 Tainted: G S
--------------------------------------------
swapper/0/1 is trying to acquire lock:
ffffffffbdc7e150 (cpu_hotplug_lock){++++}-{0:0}, at: static_key_slow_dec (kernel/jump_label.c:321 kernel/jump_label.c:336)
but task is already holding lock:
ffffffffbdc7e150 (cpu_hotplug_lock){++++}-{0:0}, at: _cpu_up (./arch/x86/include/asm/bitops.h:227 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/cpumask.h:570 ./include/linux/cpumask.h:1131 kernel/cpu.c:1641)
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(cpu_hotplug_lock);
lock(cpu_hotplug_lock);
*** DEADLOCK ***
May be due to missing lock nesting notation
2 locks held by swapper/0/1:
#0: ffffffffbdc7e058 (cpu_add_remove_lock){+.+.}-{4:4}, at: cpu_up (kernel/cpu.c:? kernel/cpu.c:1712)
#1: ffffffffbdc7e150 (cpu_hotplug_lock){++++}-{0:0}, at: _cpu_up (./arch/x86/include/asm/bitops.h:227 ./arch/x86/include/asm/bitops.h:239 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/cpumask.h:570 ./include/linux/cpumask.h:1131 kernel/cpu.c:1641)
stack backtrace:
CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Tainted: G S 6.14.0-rc5+ #13
Tainted: [S]=CPU_OUT_OF_SPEC
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 02/02/2022
Call Trace:
<TASK>
dump_stack_lvl (lib/dump_stack.c:122)
print_deadlock_bug (kernel/locking/lockdep.c:3041)
__lock_acquire (kernel/locking/lockdep.c:? kernel/locking/lockdep.c:3893 kernel/locking/lockdep.c:5228)
? asm_sysvec_apic_timer_interrupt (./arch/x86/include/asm/idtentry.h:702)
? free_one_page (mm/page_alloc.c:?)
? static_key_slow_dec (kernel/jump_label.c:321 kernel/jump_label.c:336)
lock_acquire (kernel/locking/lockdep.c:5851)
? static_key_slow_dec (kernel/jump_label.c:321 kernel/jump_label.c:336)
cpus_read_lock (./include/linux/percpu-rwsem.h:51)
? static_key_slow_dec (kernel/jump_label.c:321 kernel/jump_label.c:336)
static_key_slow_dec (kernel/jump_label.c:321 kernel/jump_label.c:336)
cond_accept_memory (mm/page_alloc.c:7024)
get_page_from_freelist (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 mm/page_alloc.c:3417)
? lock_release (kernel/locking/lockdep.c:469)
__alloc_frozen_pages_noprof (mm/page_alloc.c:4740)
__alloc_pages_noprof (mm/page_alloc.c:4774)
rb_allocate_cpu_buffer (kernel/trace/ring_buffer.c:2235)
? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/irqflags.h:42 ./arch/x86/include/asm/irqflags.h:97 ./arch/x86/include/asm/irqflags.h:155 ./include/linux/spinlock_api_smp.h:151 kernel/locking/spinlock.c:194)
trace_rb_cpu_prepare (kernel/trace/ring_buffer.c:7322)
cpuhp_invoke_callback (kernel/cpu.c:216)
? __pfx_trace_rb_cpu_prepare (kernel/trace/ring_buffer.c:7297 kernel/trace/ring_buffer.c:7304)
_cpu_up (kernel/cpu.c:967 kernel/cpu.c:990 kernel/cpu.c:1021 kernel/cpu.c:1691)
cpu_up (kernel/cpu.c:473 kernel/cpu.c:1725)
cpuhp_bringup_mask (kernel/cpu.c:1789)
? kernel_init (init/main.c:1459)
smp_init (./include/linux/bitmap.h:445 ./include/linux/nodemask.h:241 ./include/linux/nodemask.h:438 kernel/smp.c:1012)
kernel_init_freeable (init/main.c:1561)
? __pfx_kernel_init (init/main.c:1455)
kernel_init (init/main.c:1459)
ret_from_fork (arch/x86/kernel/process.c:148)
? __pfx_kernel_init (init/main.c:1455)
ret_from_fork_asm (arch/x86/entry/entry_64.S:257)
</TASK>
--
Kiryl Shutsemau / Kirill A. Shutemov