RE: [RFC PATCH v2 0/5] mm/vmscan: adaptive multi-threaded kswapd for NUMA-aware reclaim

[Ababneh, Ehab]

Hi all,

I would appreciate your thoughts on how we might achieve a similar
performance improvement without increasing the number of kswapd
threads.

In the previous patchset, using four kswapd threads reduced Cassandra
p99 latency from 6.5 ms to 3.35 ms (about 48%) and improved throughput
by 10.3%. I believe this is a substantial result and worth pursuing
further.

To explore alternatives, I tried increasing the reclaim target for
kswapd passes (testing multipliers from 2 to 100), shortening kswapd's
sleep intervals, and making it more patient before declaring a node
hopeless. So far, none of these approaches has produced a comparable
improvement.

I would be grateful for suggestions on other approaches worth
investigating.

Thanks,
Ehab


> -----Original Message-----
> From: Ababneh, Ehab
> Sent: Wednesday, October 14, 2026 12:29 AM
> To: 'Barry Song' <baohua@xxxxxxxxxx>; Shakeel Butt
> <shakeel.butt@xxxxxxxxx>; Johannes Weiner <hannes@xxxxxxxxxxx>
> Cc: ziy@xxxxxxxxxx; linux-mm@xxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> buddy.lumpkin@xxxxxxxxxx; akpm@xxxxxxxxxxxxxxxxxxxx;
> kasong@xxxxxxxxxxx; qi.zheng@xxxxxxxxx; shakeel.butt@xxxxxxxxx;
> axelrasmussen@xxxxxxxxxx; yuanchu@xxxxxxxxxx; weixugc@xxxxxxxxxx;
> david@xxxxxxxxxx; ljs@xxxxxxxxxx; liam@xxxxxxxxxxxxx; vbabka@xxxxxxxxxx;
> rppt@xxxxxxxxxx; surenb@xxxxxxxxxx; mhocko@xxxxxxxx;
> jackmanb@xxxxxxxxxx; hannes@xxxxxxxxxxx; baoquan.he@xxxxxxxxx;
> baolin.wang@xxxxxxxxxxxxxxxxx
> Subject: RE: [RFC PATCH v2 0/5] mm/vmscan: adaptive multi-threaded
> kswapd for NUMA-aware reclaim
> 
> Hi Barry, All,
> 
> > -----Original Message-----
> > From: Barry Song <baohua@xxxxxxxxxx>
> > Sent: Saturday, October 10, 2026 1:19 AM
> > To: Ababneh, Ehab <ehab.ababneh@xxxxxxxxx>
> > Cc: ziy@xxxxxxxxxx; linux-mm@xxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> > buddy.lumpkin@xxxxxxxxxx; akpm@xxxxxxxxxxxxxxxxxxxx;
> > kasong@xxxxxxxxxxx; qi.zheng@xxxxxxxxx; shakeel.butt@xxxxxxxxx;
> > axelrasmussen@xxxxxxxxxx; yuanchu@xxxxxxxxxx; weixugc@xxxxxxxxxx;
> > david@xxxxxxxxxx; ljs@xxxxxxxxxx; liam@xxxxxxxxxxxxx;
> > vbabka@xxxxxxxxxx; rppt@xxxxxxxxxx; surenb@xxxxxxxxxx;
> > mhocko@xxxxxxxx; jackmanb@xxxxxxxxxx; hannes@xxxxxxxxxxx;
> > baoquan.he@xxxxxxxxx; baolin.wang@xxxxxxxxxxxxxxxxx
> > Subject: Re: [RFC PATCH v2 0/5] mm/vmscan: adaptive multi-threaded
> > kswapd for NUMA-aware reclaim
> >
> > On Sat, Oct 10, 2026 at 4:30 AM Ehab Ababneh
> <ehab.ababneh@xxxxxxxxx>
> > wrote:
> > >
> > > This is a follow-up to the RFC for adaptive multi-threaded kswapd on
> > > NUMA. It handles concurrent MGLRU aging, adjusts wakeups to node CPU
> > > load, and scales the hopeless threshold for multiple workers.
> > >
> > > Under memory pressure, an lruvec can end up with only two MGLRU
> > > generations. That leaves an older and a younger cohort, a bit like
> > > inactive and active LRU lists, though MGLRU tracks age differently.
> > >
> > > Regarding the concerns raised in review:
> > > - In the updated Cassandra comparison, four kswapd threads reduced p99
> > >   latency from 6.5 ms to 3.35 ms (about 49%) and increased throughput by
> > >   10.3%. We believe the latency gain comes from the reclaim design,
> > >   not from changes to LRU locking.
> > > - This newer result compares four kswapd threads with one. The earlier
> > >   7.3% throughput result was measured with eight threads.
> > >
> > > We also saw periods with zero file scans. The MIN_NR_GENS guard may
> > > contribute: when only two generations remain, it makes scan_folios()
> > > return without scanning the oldest one. This patch tests whether
> > > letting kswapd scan that generation helps reclaim progress. The
> > > current traces do not tie the zero-scan events to the same lruvec,
> > > so this remains a hypothesis; more per-lruvec tracing is needed to
> > > compare it with lock contention.
> >
> > Hi Ehab,
> >
> > There is a patchset under development by Kairui, me and gaofei:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?
> > h=kairu
> > i_scanbalance_0.3
> >
> > which allows reclamation after we arrive MIN_NR_GENS.
> >
> > This might fix your problem. Hopefully, we can find some time next
> > week to send it out.
> >
> Thank you for the information.
> 
> I compared the two patch sets and here is a summary of the results:
> 
> Baseline: Linux v7.3-rc7 plus the revert of 6cbdd9726fb5.
> 
> My approach: Adaptive multi-threaded kswapd (4 threads/node), plus a
> patch that allows scanning pages when only two generations remain.
> 
> Compared against: Kairui’s scan-balance patches, based on the same baseline:
> https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kair
> ui_scanbalance_0.3
> 
> Both benchmarks used the same 12-minute Cassandra workload with 100
> client threads. Compared with the scan-balance run, my approach produced:
> 
> Throughput: 71,577 vs. 66,246 op/s, an 8.0% increase.
> p99 latency: 3.6 vs. 6.3 ms, a 42.9% reduction.
> PSI some: 41.9 vs. 109.6 s, a 61.8% reduction.
> PSI full: 39.0 vs. 97.9 s, a 60.2% reduction.
> 
> I also submitted an RFC related to regression introduce by 6cbdd9726fb5:
> https://lore.kernel.org/linux-mm/20260901180704.168106-1-
> ehab.ababneh@xxxxxxxxx/
> 
> If possible, I would like to get some tips from Shakeel or Johannes.
> 
> Shakeel and Johannes, when you have a chance, I’d appreciate your thoughts
> on the proposed patch set.
> 
> Best,
> Ehab
> 
> > Thanks
> > Barry