Re: [PATCH 13/13] percpu: queue the balance work on an unbound workqueue
From: Lorenzo Stoakes (ARM)
Date: Thu Nov 12 2026 - 15:33:10 EST
On Thu, Nov 12, 2026 at 01:41:15PM -0500, Gregory Price (Meta) wrote:
> On 2026-11-11 18:13 +0000, Lorenzo Stoakes (ARM) wrote:
> > From: Chris Mason <mason@xxxxxxxxxx>
> >
> > A burst of small atomic per-CPU allocations can fail with -ENOMEM while the
> > machine has free memory and idle CPUs, as atomic per-CPU allocations are
> > only served from pages that are already populated, and those can be used up
> > before a worker gets to populate more.
> >
> > Consider:
> >
> > caller on CPU 1 kworker bound to CPU 1
> > =============== ======================
> > allocates 8 bytes, which leaves
> > one empty page and 4088 bytes
> > schedule_work() woken, waits for CPU 1
> > allocates 8 bytes, 1023 times
> > next allocation: "no space left"
> > gives up CPU 1 populates pages
> >
> > The allocator keeps a few empty populated pages in reserve, and the work
> > item pcpu_balance_work refills them.
> >
> > pcpu_alloc_noprof() queues the work item using schedule_work(), so only a
> > worker bound to the CPU of the caller can run the work item.
> >
> > The issue is that the caller keeps that CPU - a task keeps it for the rest
> > of its time slice, and a softirq keeps it until the softirq ends.
> >
> > In that time the caller can use up the reserve, resulting in an atomic
> > per-CPU allocation failing.
> >
> > Fix the issue by queueing the work item on an unbound workqueue so that
> > another CPU can populate pages while the caller keeps allocating.
> >
> > Fixes: 1a4d76076cda ("percpu: implement asynchronous chunk population")
> > Signed-off-by: Chris Mason <mason@xxxxxxxxxx>
> > [ ljs: tweaked commit message, comment ]
> > Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
>
> Reproduced with a cpu hammering atomic percpu allocations with preempt
> disabled. Results are pretty cool.
>
> In about 1 second of hammering allocations with the full 13 patch series:
>
> Base Patch
> Successful Allocs 1,188 131,072
> Failed allocs 4,097 0
> balance worker runs 0 169
> successful retries 0 4,096
>
> Micro-bench obviously, but pretty stark difference.
>
> Tested-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
> Reviewed-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
Thanks, that's great!
I will include that in commit msg on respin.
>
> --
> ~Gregory
>
--
Cheers, Lorenzo