Re: [PATCH v2 13/13] percpu: queue the balance work on an unbound workqueue
From: Lorenzo Stoakes (ARM)
Date: Sun Nov 15 2026 - 07:41:17 EST
On Sun, Nov 15, 2026 at 10:07:15AM +1100, Balbir Singh wrote:
> On Fri, Nov 13, 2026 at 09:45:31PM +0000, Lorenzo Stoakes (ARM) wrote:
> > From: Chris Mason <mason@xxxxxxxxxx>
> >
> > A burst of small atomic per-CPU allocations can fail with -ENOMEM while the
> > machine has free memory and idle CPUs, as atomic per-CPU allocations are
> > only served from pages that are already populated, and those can be used up
> > before a worker gets to populate more.
> >
> > Consider:
> >
> > caller on CPU 1 kworker bound to CPU 1
> > =============== ======================
> > allocates 8 bytes, which leaves
> > one empty page and 4088 bytes
> > schedule_work() woken, waits for CPU 1
> > allocates 8 bytes, 1023 times
> > next allocation: "no space left"
> > gives up CPU 1 populates pages
> >
> > The allocator keeps a few empty populated pages in reserve, and the work
> > item pcpu_balance_work refills them.
> >
> > pcpu_alloc_noprof() queues the work item using schedule_work(), so only a
> > worker bound to the CPU of the caller can run the work item.
> >
> > The issue is that the caller keeps that CPU - a task keeps it for the rest
> > of its time slice, and a softirq keeps it until the softirq ends.
> >
> > In that time the caller can use up the reserve, resulting in an atomic
> > per-CPU allocation failing.
> >
> > Fix the issue by queueing the work item on an unbound workqueue so that
> > another CPU can populate pages while the caller keeps allocating.
> >
> > Gregory Price hammered atomic per-CPU allocations from one CPU with
> > preemption disabled for about a second, with and without this series:
> >
> > Before After
> > Successful allocs 1,188 131,072
> > Failed allocs 4,097 0
> > Balance worker runs 0 169
> > Successful retries 0 4,096
> >
> > Fixes: 1a4d76076cda ("percpu: implement asynchronous chunk population")
> > Signed-off-by: Chris Mason <mason@xxxxxxxxxx>
> > [ ljs: tweaked commit message, comment ]
> > Tested-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
> > Reviewed-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
> > Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
> > ---
> > mm/percpu.c | 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/mm/percpu.c b/mm/percpu.c
> > index f87813e6ad73..39cba1710383 100644
> > --- a/mm/percpu.c
> > +++ b/mm/percpu.c
> > @@ -200,8 +200,9 @@ static bool pcpu_atomic_alloc_failed;
> >
> > static void pcpu_schedule_balance_work(void)
> > {
> > + /* Unbound, so another CPU can populate pgs if caller keeps this one. */
> > if (pcpu_async_enabled)
> > - schedule_work(&pcpu_balance_work);
> > + queue_work(system_dfl_wq, &pcpu_balance_work);
> > }
> >
>
> Can this run havoc on otherwise idle systems? Why not choose a workqueue
I don't think so - this doesn't change when or how often the work is
queued, just _where_ it can run and it only happens when the allocator dips
below the empty page low watermark, an atomic alloc fails or there's
something to reclaim.
Also unbound workqueues default to cache-local affinity, and obey the
unbound cpumask so stays off isolated cores. So that's actually rather an
improvement on happening to run on whichever core allocated :)
> with WQ_POWER_EFFICIENT, which becomes unbound when nohz_full is
> enabled? I guess it still leaves open the case when nohz_full is not
> enabled and we get a burst of atomic allocations?
Yeah exactly. We'd just end up not fixing the bug otherwise.
--
Cheers, Lorenzo