Re: Improve page refcount scalability

[Linus Torvalds]

On Tue, 13 Oct 2026 at 17:10, Harry Yoo <harry@xxxxxxxxxx> wrote:
>
> There should be a limitation on how much memory can be submitted to RCU
> by the page cache, otherwise will be a corner case where we have too
> much unreclaimable memory waiting for the RCU subsystem to free them.

There is. In fact, I tried it a few different ways... It's not quite
what you suggest, but try it out.

> For the same reason, when a process tries to create RCU-delay-freed
> pages much faster than RCU can free memory, the kernel needs to limit
> that.

That was actually my initial approach - making the places that set the
PG_rcu_delay bit throttle themselves when there were lots of such
pages in flight.

Except I worried about slowing down simple things like truncating big
files: particularly since I believe firmly that slowing that down is
the *wrong* thing to do from a memory pressure standpoint, because
you'd be slowing down the very thing that is actively freeing pages -
it's just that it then takes a (short) while for those pages to get to
a usable state.

So what I did instead was to actively then expedite RCU when there
were a lot of pages in flight.

That worked quite well, but it hit a problem when you had lots of CPUs
doing it: since an RCU quiet period needs to happen on *each* CPU, you
still end up wanting to at some point wait for things: you need to
synchronize not _one_ thread freeing things, but synchronize across
all the CPU's.

But again: the good part really is that this "lots and lots of pages
in flight" only happens when there are literally lots and lots of
pages that are very much about to be free'd. So you are in a fairly
good position when it happens, you just need to *understand* the
position you are in.

So the other part is to come at the issue from the other end: when you
have trouble allocating pages, but see that there are a ton of pages
in flight due to PG_rcu_free, you basically do the same thing from the
other end, and do synchronize_rcu_expedited().

And in testing, that worked really well under even insane loads where
concurrent hole punching or truncate calls would generate gigabytes of
PG_rcu_free activity each second.

Now, it turns out that Kiryl also suggested another approach to the
pages becoming PG_rcu_free in the first place: instead of being able
to easily trigger gigabytes of such free pages by just truncating a
large file - which marked all those pages PG_rcu_free whether that was
useful or not - the current version actually comes at *that* the other
way too: mark only pages that are hot and are *worth* the PG_rcu_free
treatment (because they get read over and over again) with PG_rcu_free
in the first place, so that you don't generate that flood of
RCU-free'd pages in the first place, and now the PG_rcu_free flood is
in a way throttled by the *use* of said pages.

So there's effectively three different things working together:

 - mark a folio PG_rcu_free only when it's hot and keeps being accessed

   (See commits dabc522cee5b: "mm: have folio_mark_accessed() return
whether the folio is hot" and  bb5908b733a1: "mm/filemap: set
PG_rcu_free on the folios that reads keep coming back to" for details
- both are pretty tiny and not really all that complicated)

 - proactively expedite RCU grace periods when there are a lot of of
pages pending

   (See commit 2ce8df4627fe: "mm: expedite the grace period when a CPU
has many deferred frees")

 - teach the page allocator about the outstanding pages ratehr that failing

   (See commit 82fc13ff3a1e mm: "count the pages waiting for a grace
period and free them before OOM")

and it really does seem to work quite smoothly. But it's not like I
can guarantee it's perfect.

All those commits are fairly trivial. Of the ones listed above, that
"count the pages waiting for a grace period and free them before OOM"
is the biggest by far, but it's not really particularly complex or
subtle.

The actual biggest patches in the series are:

 - 65401f4102b9 ("mm/filemap: read from the page cache without folio
references")

   This really is not very big or complicated, although it *does* have
subtle ordering issues with i_size as pointed out by Kiryl. It's
mostly really just a simplified version of my patch from two and a
half years ago. And it's simplified because now the RCU freeing makes
everything more natural.

 - df0b133495c9 ("mm: free PG_rcu_free folios only after an RCU grace period")

   This is the one that actually runs on the page freeing side and
adds a page that is marked PG_rcu_free to a list and sets up the whole
RCU machinery (ok, there are actually two per-cpu lists: the 'pending'
list and the 'batch' list, and a RCU grace period will free the
current batch and move the pending list to be the next batch - not
rocket science).

and even those "big" patches aren't really all that big. Both are less
than 200 lines of code, and while I wouldn't call either of them
_trivial_, both are very much doing one clear thing and there's
nothing really *strange* about them either.

So it's ten patches, none of them particularly hairy. I really would
encourage you to just take a look and take it for a try.

Now, I *will* say that obviously RCU-freeing pages is an absolutely
conceptually *enormous* change to the VM. But it's in no way at all a
new thing: we've been doing the same thing for kmalloc() for decades.
With the right pattern, you can cause a lot of RCU-delayed kfree
loads. And we've had to tweak things for that.

So do I believe that branch is perfect on the first try? Hell no. I'm
sure there are things to improve there. But there are few syustem
calls more important than 'read()', so I think it's worth doing. We've
been doing the RCU-freeing for almost every other path, and I will
argue very forcefully that the path handling RCU code is an order of
magnitude more complicated than this particular code is.

              Linus