Re: [PATCH 13/13] percpu: queue the balance work on an unbound workqueue

[Gregory Price (Meta)]

On 2026-11-11 18:13 +0000, Lorenzo Stoakes (ARM) wrote:
> From: Chris Mason <mason@xxxxxxxxxx>
> 
> A burst of small atomic per-CPU allocations can fail with -ENOMEM while the
> machine has free memory and idle CPUs, as atomic per-CPU allocations are
> only served from pages that are already populated, and those can be used up
> before a worker gets to populate more.
> 
> Consider:
> 
>     caller on CPU 1                     kworker bound to CPU 1
>     ===============                     ======================
>     allocates 8 bytes, which leaves
>     one empty page and 4088 bytes
>       schedule_work()                   woken, waits for CPU 1
>     allocates 8 bytes, 1023 times
>     next allocation: "no space left"
>     gives up CPU 1                      populates pages
> 
> The allocator keeps a few empty populated pages in reserve, and the work
> item pcpu_balance_work refills them.
> 
> pcpu_alloc_noprof() queues the work item using schedule_work(), so only a
> worker bound to the CPU of the caller can run the work item.
> 
> The issue is that the caller keeps that CPU - a task keeps it for the rest
> of its time slice, and a softirq keeps it until the softirq ends.
> 
> In that time the caller can use up the reserve, resulting in an atomic
> per-CPU allocation failing.
> 
> Fix the issue by queueing the work item on an unbound workqueue so that
> another CPU can populate pages while the caller keeps allocating.
> 
> Fixes: 1a4d76076cda ("percpu: implement asynchronous chunk population")
> Signed-off-by: Chris Mason <mason@xxxxxxxxxx>
> [ ljs: tweaked commit message, comment ]
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>

Reproduced with a cpu hammering atomic percpu allocations with preempt
disabled. Results are pretty cool.

In about 1 second of hammering allocations with the full 13 patch series:

                        Base       Patch
Successful Allocs      1,188      131,072 
Failed allocs          4,097         0
balance worker runs      0          169     
successful retries       0         4,096

Micro-bench obviously, but pretty stark difference.

Tested-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
Reviewed-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>

-- 
~Gregory