Re: [RFC 2/2] mm: page_alloc: per-cpu pageblock buddy allocator

[Yafang Shao]

On Tue, Sep 22, 2026 at 4:03 AM Johannes Weiner <hannes@xxxxxxxxxxx> wrote:
>
> Hello Yafang,
>
> On Fri, Sep 18, 2026 at 10:22:22AM +0800, Yafang Shao wrote:

[...]

> >
> > Hello Johannes,
> >
> > Thank you for the great work on this series -- I hope it is still being
> > actively worked on.
>
> Thanks for the kind words.
>
> I am still actively working on it. Since the last iteration I have
> addressed a few things:
>
> 1. The locking bug you are seeing. Rik had also run into this during
> stress testing. The fallback to the zone buddy on PCP contention
> brought back some of the original zone->lock contention.

Right, I noticed that the zone::lock contention increased slightly
after I applied the livepatch above.

> So instead
> I'm using the zone llist introduced for lockless allocations.

That is good news!

>
> 2. The PFN search for block recovery that Vlastimil pointed out. I've
> tried various solutions (counters, bitmaps) but the thing that worked
> best was having the zone buddy itself maintain free pages of owned
> blocks on a per-block loaner list (in addition to the regular zone
> freelists). This eliminates the sparse search altogether. Recovery is
> then: pcp->owned_blocks -> pbd->buddy_loans -> page. Every page
> visited gets recovered. For the loaner list_head, I'm reusing
> mapping/index space that's unused in a freed page.
>
> 3. Removed the unowned buddy splitting on the PCP. Vlastimil had
> actually asked to try that separately, as an incremental step, since
> it's self contained. I tried this but realized that part was actually
> bad altogether. It violates the rmqueue_smallest policy and causes
> runaway fragmentation - just like the new block claiming did before I
> added the block recovery step beforehand.
>
> So now refilling is just block recovery -> new blocks -> unowned
> singles of the requested order.
>
> Incidentally, this also eliminated the CMA problem that Frank pointed
> out, since the other refill paths respect ALLOC_CMA.
>
> 4. I realized I'm also violating the smallest-first policy in how I
> was mixing owned and unowned chunks on the same freelists. For
> example, an order-3 refill from singles sits next to order-3 fragments
> from owned blocks. Only owned fragments, which route back to and
> reassemble on that PCP, must be split. Unowned singles must be
> consumed at their native order to preserve smallest-first policy.
>
> pcp_rmqueue_smallest() could check the PagePCPBuddy() flag to tell
> which ones can be split, but that introduces another sparse search
> problem, where we might walk higher order lists in the hope to find a
> splittable owned buddy.
>
> To avoid this, I retained the legacy/unowned pcp freelists (up to
> costly order and THP), and added a second set of buddy freelists up to
> pageblock order to the PCP. This way the rule can be maintained with
> O(1) list checks instead of O(pcp size) scans.
>
> 5. The on-demand merging at drain time proved problematic. Draining
> isn't exhaustive, so it can attempt to merge the same unmergeable
> fragments repeatedly. I moved merging into the pcp free path instead,
> so every page is tried for merging exactly once, which seems to
> perform a lot better in performance testing.
>
> Overall, it's gotten a bit bigger than I had hoped for. But it also
> looks much more robust. And the additions described above seem well
> offset by performance improvements in tests so far, even on smaller
> machines.

It will take me some time to fully understand your work, but thank you
so much for your great effort in resolving this long-standing
zone::lock contention issue.

>
> I'm still testing and polishing right now, and hoping to send a new
> version soon.

Great

>
> > We are suffering from heavy zone->lock contention on our production
> > servers as well, so I backported this series to our internal 6.18.y
> > kernel. However, since deploying it to a few dozen production servers
> > running workloads with heavy memory and I/O pressure, we have been
> > hitting hard lockups at a rate of roughly one every day or two. The
> > hard lockups look as follows:
>
> [...]
>
> > With these changes applied, the affected servers have been running
> > lockup-free for more than two weeks so far.
>
> I'm assuming you saw an improvement of zone->lock contention. Would
> you be able to share some numbers or observations?

The result is great.  Below are the observations from our real
production workloads.

On our Kubernetes servers, many different applications (Pods) are
deployed on a single host, and redis is very latency-sensitive.
Whenever other Pods exit simultaneously, or any Pod allocates or frees
a large amount of memory, redis easily hits latency spikes, and its
P99 can quickly reach a level the application can't tolerate.

This issue is clearly reflected by the TCPTimeouts metric in
/proc/net/netstat, since the network receive path also needs to hold
the zone::lock. Before your improvement, Redis's TCPTimeouts could
reach tens or even hundreds whenever other Pods exited simultaneously
or allocated/freed large amounts of memory. With your improvement
applied, TCPTimeouts has stayed below 10 and Redis's P99 has remained
acceptable. After I deployed the hotfix to work around the hard lockup
issue, TCPTimeouts rose slightly, but it is still far lower than
before your improvement.

-- 
Regards
Yafang