Re: Improve page refcount scalability

[Paul E. McKenney]

On Tue, Oct 13, 2026 at 10:03:12PM +0100, Matthew Wilcox wrote:
> On Tue, Oct 13, 2026 at 11:43:14AM -0700, Linus Torvalds wrote:
> > The fairly short series of ten patches basically enhances folio lookup
> > by noticing when folios are hot, and setting a "delay relasing this
> > folio until after a RCU period", and then using that RCU delay it does
> > not just the folio lookup itself under RCU (which we've done forever),
> > but also the actual folio data accesses themselves under RCU. No
> > refcount taking needed for the fastpath where we can just read the
> > data from the folio entirely in an RCU-protected region when the user
> > copy doesn't cause page faults.
> > 
> > Unlike my original approach - because it now actually delays the
> > freeing of the underlying folio data - there's no need for any double
> > buffering, and a lot of the original limitations of my original patch
> > go away too. That improves performance quite a lot. And by "quite a
> > lot"  I mean "I see a 23x performance increase on my Ryzen
> > threadripper". Admittedly that is a very artificial case of reading
> > the same small part of a page over and over in a threaded loop on all
> > CPUs, so any real life impact will be much much less.
> > 
> > Anyway, I wasn't planning on posting the individual patches, I think
> > we all use git by now and probably find it easier than fetching
> > patches from the list. So it's all at
> > 
> >     git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux rcu-free-v3
> > 
> > and the commit messages are a bit more verbose than I would have done
> > it myself without tool use, but maybe it's better that way.
> 
> Obviously the numbers are compelling.  And thanks to Kiryl for figuring
> out a workload that confirmed my fears, then remedying the problem.
> 
> I don't like the PG_rcu_free flag.  I hope we're guaranteed that folios
> in the process of being freed by RCU will not be mapped into userspace.
> That means we can use PGTY_rcu instead, where there is no shortage of
> available types.
> 
> The workload that Kiryl identified (truncating/holepunching folios
> from files) is not the normal way we free folios from the page cache,
> so while we need to accommodate it, it's not what we shuld optimise for.
> Most page cache folios get freed through reclaim.  I started sketching
> out a new state (I was going to call them "ghost folios") when I realised
> I was basically reinventing the inactive list.  So maybe we should make
> folios on the inactive list immediately freeable as long as one RCU
> generation has passed?
> 
> This may be where I run into trouble.  I don't know how RCU works
> internally.  Is it possible to record some kind of generation number in
> the folio such that we can see if it's only recently been added to the
> inactive list?  Or do we need three lists --active, purgatory, inactive
> where the folio is moved from active to purgatory as it ages, then
> from purgatory to inactive after an RCU perid?

The polled grace-period APIs handle the last sentence for you, but you
knew that already.  I am not sure what "recently added" means in this
context, but I cannot resist suggesting using the jiffies counter.

But tell me more about the meaning of "recently added" and I will try
to be more helpful.  ;-)

							Thanx, Paul

> This will somewhat complicate filemap_get_entry().  It'll have to be
> able to cope with finding folios in all three states, and moving
> them from the inactive or purgatory lists back to the active list.
> I think it's doable with the right cmpxchg() magic.
> 
> What do you think?