[PATCH v2 13/13] percpu: queue the balance work on an unbound workqueue

From: Lorenzo Stoakes (ARM)

Date: Fri Nov 13 2026 - 16:48:06 EST


From: Chris Mason <mason@xxxxxxxxxx>

A burst of small atomic per-CPU allocations can fail with -ENOMEM while the
machine has free memory and idle CPUs, as atomic per-CPU allocations are
only served from pages that are already populated, and those can be used up
before a worker gets to populate more.

Consider:

caller on CPU 1 kworker bound to CPU 1
=============== ======================
allocates 8 bytes, which leaves
one empty page and 4088 bytes
schedule_work() woken, waits for CPU 1
allocates 8 bytes, 1023 times
next allocation: "no space left"
gives up CPU 1 populates pages

The allocator keeps a few empty populated pages in reserve, and the work
item pcpu_balance_work refills them.

pcpu_alloc_noprof() queues the work item using schedule_work(), so only a
worker bound to the CPU of the caller can run the work item.

The issue is that the caller keeps that CPU - a task keeps it for the rest
of its time slice, and a softirq keeps it until the softirq ends.

In that time the caller can use up the reserve, resulting in an atomic
per-CPU allocation failing.

Fix the issue by queueing the work item on an unbound workqueue so that
another CPU can populate pages while the caller keeps allocating.

Gregory Price hammered atomic per-CPU allocations from one CPU with
preemption disabled for about a second, with and without this series:

Before After
Successful allocs 1,188 131,072
Failed allocs 4,097 0
Balance worker runs 0 169
Successful retries 0 4,096

Fixes: 1a4d76076cda ("percpu: implement asynchronous chunk population")
Signed-off-by: Chris Mason <mason@xxxxxxxxxx>
[ ljs: tweaked commit message, comment ]
Tested-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
Reviewed-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
---
mm/percpu.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/percpu.c b/mm/percpu.c
index f87813e6ad73..39cba1710383 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -200,8 +200,9 @@ static bool pcpu_atomic_alloc_failed;

static void pcpu_schedule_balance_work(void)
{
+ /* Unbound, so another CPU can populate pgs if caller keeps this one. */
if (pcpu_async_enabled)
- schedule_work(&pcpu_balance_work);
+ queue_work(system_dfl_wq, &pcpu_balance_work);
}

/**

--
2.56.0