Re: [PATCH v2] mm: let kswapd work again for node that used to be hopeless but may not now

[Huang, Ying]

Byungchul Park <byungchul@xxxxxx> writes:

> On Wed, Jun 05, 2024 at 08:59:01AM +0800, Huang, Ying wrote:
>> Byungchul Park <byungchul@xxxxxx> writes:
>> 
>> > On Tue, Jun 04, 2024 at 08:29:27AM -0400, Johannes Weiner wrote:
>> >> On Tue, Jun 04, 2024 at 07:25:16PM +0900, Byungchul Park wrote:
>> >> > On Tue, Jun 04, 2024 at 06:12:22PM +0900, Byungchul Park wrote:
>> >> > > On Tue, Jun 04, 2024 at 04:57:17PM +0800, Huang, Ying wrote:
>> >> > > > Byungchul Park <byungchul@xxxxxx> writes:
>> >> > > > 
>> >> > > > > On Tue, Jun 04, 2024 at 03:57:54PM +0800, Huang, Ying wrote:
>> >> > > > >> Byungchul Park <byungchul@xxxxxx> writes:
>> >> > > > >> 
>> >> > > > >> > Changes from v1:
>> >> > > > >> > 	1. Don't allow to resume kswapd if the system is under memory
>> >> > > > >> > 	   pressure that might affect direct reclaim by any chance, like
>> >> > > > >> > 	   if NR_FREE_PAGES is less than (low wmark + min wmark)/2.
>> >> > > > >> >
>> >> > > > >> > --->8---
>> >> > > > >> > From 6c73fc16b75907f5da9e6b33aff86bf7d7c9dd64 Mon Sep 17 00:00:00 2001
>> >> > > > >> > From: Byungchul Park <byungchul@xxxxxx>
>> >> > > > >> > Date: Tue, 4 Jun 2024 15:27:56 +0900
>> >> > > > >> > Subject: [PATCH v2] mm: let kswapd work again for node that used to be hopeless but may not now
>> >> > > > >> >
>> >> > > > >> > A system should run with kswapd running in background when under memory
>> >> > > > >> > pressure, such as when the available memory level is below the low water
>> >> > > > >> > mark and there are reclaimable folios.
>> >> > > > >> >
>> >> > > > >> > However, the current code let the system run with kswapd stopped if
>> >> > > > >> > kswapd has been stopped due to more than MAX_RECLAIM_RETRIES failures
>> >> > > > >> > until direct reclaim will do for that, even if there are reclaimable
>> >> > > > >> > folios that can be reclaimed by kswapd.  This case was observed in the
>> >> > > > >> > following scenario:
>> >> > > > >> >
>> >> > > > >> >    CONFIG_NUMA_BALANCING enabled
>> >> > > > >> >    sysctl_numa_balancing_mode set to NUMA_BALANCING_MEMORY_TIERING
>> >> > > > >> >    numa node0 (500GB local DRAM, 128 CPUs)
>> >> > > > >> >    numa node1 (100GB CXL memory, no CPUs)
>> >> > > > >> >    swap off
>> >> > > > >> >
>> >> > > > >> >    1) Run a workload with big anon pages e.g. mmap(200GB).
>> >> > > > >> >    2) Continue adding the same workload to the system.
>> >> > > > >> >    3) The anon pages are placed in node0 by promotion/demotion.
>> >> > > > >> >    4) kswapd0 stops because of the unreclaimable anon pages in node0.
>> >> > > > >> >    5) Kill the memory hoggers to restore the system.
>> >> > > > >> >
>> >> > > > >> > After restoring the system at 5), the system starts to run without
>> >> > > > >> > kswapd.  Even worse, tiering mechanism is no longer able to work since
>> >> > > > >> > the mechanism relies on kswapd for demotion.
>> >> > > > >> 
>> >> > > > >> We have run into the situation that kswapd is kept in failure state for
>> >> > > > >> long in a multiple tiers system.  I think that your solution is too
>> >> > > > >
>> >> > > > > My solution just gives a chance for kswapd to work again even if
>> >> > > > > kswapd_failures >= MAX_RECLAIM_RETRIES, if there are potential
>> >> > > > > reclaimable folios.  That's it.
>> >> > > > >
>> >> > > > >> limited, because OOM killing may not happen, while the access pattern of
>> >> > > > >
>> >> > > > > I don't get this.  OOM will happen as is, through direct reclaim.
>> >> > > > 
>> >> > > > A system that fails to reclaim via kswapd may succeed to reclaim via
>> >> > > > direct reclaim, because more CPUs are used to scanning the page tables.
>> >> > > > 
>> >> > > > In a system with NUMA balancing based page promotion and page demotion
>> >> > > > enabled, page promotion will wake up kswapd, but kswapd may fail in some
>> >> > > > situations.  But page promotion will no trigger direct reclaim or OOM.
>> >> > > > 
>> >> > > > >> the workloads may change.  We have a preliminary and simple solution for
>> >> > > > >> this as follows,
>> >> > > > >> 
>> >> > > > >> https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/commit/?h=tiering-0.8&id=17a24a354e12d4d4675d78481b358f668d5a6866
>> >> > > > >
>> >> > > > > Whether tiering is involved or not, the same problem can arise if
>> >> > > > > kswapd gets stopped due to kswapd_failures >= MAX_RECLAIM_RETRIES.
>> >> > > > 
>> >> > > > Your description is about tiering too.  Can you describe a situation
>> >> > > 
>> >> > > I mentioned "tiering" while I described how to reproduce because I ran
>> >> > > into the situation while testing with tiering system but I don't think
>> >> > > it's the necessary condition.
>> >> > > 
>> >> > > Let me ask you back, why the logic to stop kswapd was considered in the
>> >> > > first place?  That's because the problem was already observed anyway
>> >> > 
>> >> > To be clear..
>> >> > 
>> >> > The problem, kswapd_failures >= MAX_RECLAIM_RETRIES, can happen whether
>> >> > tiering is involved not not.  Once kswapd stops, the system should run
>> >> > without kswapd even after recovered e.g. by killing the hoggers.  *Even
>> >> > worse*, tiering mechanism doesn't work in this situation.
>> >> 
>> >> But like Ying said, in other situations it's direct reclaim that kicks
>> >> in and clears the flag.
>> >
>> > I already described it in the commit message.
>> >
>> >> The failure-sleep and direct reclaim triggered recovery have been in
>> >
>> > Sure.  It's better than nothing.
>> >
>> >> place since 2017. Both parties who observed an issue with it recently
>> >> did so in tiering scenarios. IMO a tiering-specific solution makes the
>> >> most sense.
>> >
>> > So..  Is the follow situation in a non-tiering system okay?  Really?
>> >
>> >    A system runs with kswapd disabled unless hitting min water mark,
>> >    even if there might be something that kswapd can work on.
>> >
>> > I don't undertand why it's okay.  Could you explain more?  Then why do
>> > we use kswapd in background?
>> 
>> IIUC, it's okey.  One direct reclaiming will be triggered, then kswapd
>> reclaiming will be recovered.  So, the performance will not be
>> influenced much.
>
> So is it because the performance will not be influenced much?  Hm..  the
> system would get impacted at the moment when direct reclaim gets
> triggerred, even though kswapd can mitigate the impact proactively.
>
> However, I don't want to insist strongly if you all consider it's okay.
>
> Changing the topic to tiering, which one looks better between two
> appoaches to solve the issue that tiering doens't work once the failures
> hit MAX_RECLAIM_RETRIES:
>
>    1) periodically run kswapd
>    2) run kswapd if there might be reclaimable folios as this patch does

It's hard to capture all situations.  Folios may become reclaimable in
various ways, some folios become cold, swap devices are added, munlock,
folios freeing/allocating, etc.  It's hard to detect them all without
trying.

It may be not necessary to run kswapd periodically.  So, another
possibility is to allow wakeup kswapd after some timeout even if
kswapd_failures >= MAX_RECLAIM_RETRIES.

--
Best Regards,
Huang, Ying

> For 2), this patch should be modified a lil bit tho.
>
> 	Byungchul
>
>> Do you think that this will impact performance?  If so, please try to
>> prove it with test results.
>> 
>> --
>> Best Regards,
>> Huang, Ying