Re: [RFC 2/2] mm: page_alloc: per-cpu pageblock buddy allocator

[Johannes Weiner]

Hello Yafang,

On Fri, Sep 18, 2026 at 10:22:22AM +0800, Yafang Shao wrote:
> On Fri,  3 Apr 2026 at 15:40 PM Johannes Weiner <hannes@xxxxxxxxxxx> wrote:
> 
> [...]
> 
> > @@ -2941,15 +3242,45 @@ static void __free_frozen_pages(struct page *page, unsigned int order,
> [...]
> > +	pcp = per_cpu_ptr(zone->per_cpu_pageset, cache_cpu);
> > +	if (unlikely(fpi_flags & FPI_TRYLOCK) || !in_task()) {
> > +		if (!spin_trylock_irqsave(&pcp->lock, UP_flags)) {
> > +			free_one_page(zone, page, pfn, order, fpi_flags);
> >  			return;
> > -		pcp_spin_unlock(pcp, UP_flags);
> > +		}
> >  	} else {
> > +		spin_lock_irqsave(&pcp->lock, UP_flags);
> > +	}
> 
> [...]
> 
> > @@ -3025,17 +3369,35 @@ void free_unref_folios(struct folio_batch *folios)
> [...]
> > +			if (!in_task()) {
> > +				if (unlikely(!spin_trylock_irqsave(
> > +						&pcp->lock, UP_flags))) {
> > +					pcp = NULL;
> > +					free_one_page(zone, &folio->page, pfn,
> > +						      order, FPI_NONE);
> > +					continue;
> > +				}
> > +			} else {
> > +				spin_lock_irqsave(&pcp->lock, UP_flags);
> > +			}
> 
> Hello Johannes,
> 
> Thank you for the great work on this series -- I hope it is still being
> actively worked on.

Thanks for the kind words.

I am still actively working on it. Since the last iteration I have
addressed a few things:

1. The locking bug you are seeing. Rik had also run into this during
stress testing. The fallback to the zone buddy on PCP contention
brought back some of the original zone->lock contention. So instead
I'm using the zone llist introduced for lockless allocations.

2. The PFN search for block recovery that Vlastimil pointed out. I've
tried various solutions (counters, bitmaps) but the thing that worked
best was having the zone buddy itself maintain free pages of owned
blocks on a per-block loaner list (in addition to the regular zone
freelists). This eliminates the sparse search altogether. Recovery is
then: pcp->owned_blocks -> pbd->buddy_loans -> page. Every page
visited gets recovered. For the loaner list_head, I'm reusing
mapping/index space that's unused in a freed page.

3. Removed the unowned buddy splitting on the PCP. Vlastimil had
actually asked to try that separately, as an incremental step, since
it's self contained. I tried this but realized that part was actually
bad altogether. It violates the rmqueue_smallest policy and causes
runaway fragmentation - just like the new block claiming did before I
added the block recovery step beforehand.

So now refilling is just block recovery -> new blocks -> unowned
singles of the requested order.

Incidentally, this also eliminated the CMA problem that Frank pointed
out, since the other refill paths respect ALLOC_CMA.

4. I realized I'm also violating the smallest-first policy in how I
was mixing owned and unowned chunks on the same freelists. For
example, an order-3 refill from singles sits next to order-3 fragments
from owned blocks. Only owned fragments, which route back to and
reassemble on that PCP, must be split. Unowned singles must be
consumed at their native order to preserve smallest-first policy.

pcp_rmqueue_smallest() could check the PagePCPBuddy() flag to tell
which ones can be split, but that introduces another sparse search
problem, where we might walk higher order lists in the hope to find a
splittable owned buddy.

To avoid this, I retained the legacy/unowned pcp freelists (up to
costly order and THP), and added a second set of buddy freelists up to
pageblock order to the PCP. This way the rule can be maintained with
O(1) list checks instead of O(pcp size) scans.

5. The on-demand merging at drain time proved problematic. Draining
isn't exhaustive, so it can attempt to merge the same unmergeable
fragments repeatedly. I moved merging into the pcp free path instead,
so every page is tried for merging exactly once, which seems to
perform a lot better in performance testing.

Overall, it's gotten a bit bigger than I had hoped for. But it also
looks much more robust. And the additions described above seem well
offset by performance improvements in tests so far, even on smaller
machines.

I'm still testing and polishing right now, and hoping to send a new
version soon.

> We are suffering from heavy zone->lock contention on our production
> servers as well, so I backported this series to our internal 6.18.y
> kernel. However, since deploying it to a few dozen production servers
> running workloads with heavy memory and I/O pressure, we have been
> hitting hard lockups at a rate of roughly one every day or two. The
> hard lockups look as follows:

[...]

> With these changes applied, the affected servers have been running
> lockup-free for more than two weeks so far.

I'm assuming you saw an improvement of zone->lock contention. Would
you be able to share some numbers or observations?

Thanks again!