Re: [PATCH 10/10] mm: balance LRU lists based on relative thrashing

[Johannes Weiner]

On Fri, Jun 17, 2016 at 04:49:45PM +0900, Minchan Kim wrote:
> On Thu, Jun 16, 2016 at 11:12:07AM -0400, Johannes Weiner wrote:
> > On Wed, Jun 15, 2016 at 11:23:41AM +0900, Minchan Kim wrote:
> > > On Mon, Jun 13, 2016 at 11:52:31AM -0400, Johannes Weiner wrote:
> > > > On Fri, Jun 10, 2016 at 11:19:35AM +0900, Minchan Kim wrote:
> > > > > Other concern about PG_workingset is naming. For file-backed pages, it's
> > > > > good because file-backed pages started from inactive's head and promoted
> > > > > active LRU once two touch so it's likely to be workingset. However,
> > > > > for anonymous page, it starts from active list so every anonymous page
> > > > > has PG_workingset while mlocked pages cannot have a chance to have it.
> > > > > It wouldn't matter in eclaim POV but if we would use PG_workingset as
> > > > > indicator to identify real workingset page, it might be confused.
> > > > > Maybe, We could mark mlocked pages as workingset unconditionally.
> > > > 
> > > > Hm I'm not sure it matters. Technically we don't have to set it on
> > > > anon, but since it's otherwise unused anyway, it's nice to set it to
> > > > reinforce the notion that anon is currently always workingset.
> > > 
> > > When I read your description firstly, I thought the flag for anon page
> > > is set on only swapin but now I feel you want to set it for all of
> > > anonymous page but it has several holes like mlocked pages, shmem pages
> > > and THP and you want to fix it in THP case only.
> > > Hm, What's the rule?
> > > It's not consistent and confusing to me. :(
> > 
> > I think you are might be over thinking this a bit ;)
> > 
> > The current LRU code has a notion of workingset pages, which is anon
> > pages and multi-referenced file pages. shmem are considered file for
> > this purpose. That's why anon start out active and files/shmem do
> > not. This patch adds refaulting pages to the mix.
> > 
> > PG_workingset keeps track of pages that were recently workingset, so
> > we set it when the page enters the workingset (activations and
> > refaults, and new anon from the start). The only thing we need out of
> > this flag is to tell us whether reclaim is going after the workingset
> > because the LRUs have become too small to hold it.
> 
> Understood.
> 
> Divergence comes from here. It seems you design the page flag for only
> aging/balancing logic working well while I am thinking to leverage the
> flag to identify real workingset. I mean a anonymous page would be a cold
> if it has just cold data for the application which would be swapped
> out after a short time and never swap-in until process exits. However,
> we put it from active list so that it has PG_workingset but it's cold
> page.
> 
> Yes, we cannot use the flag for such purpose in this SEQ replacement so
> I will not insist on it.

Well, I'm designing the flag so that it's useful for the case I am
introducing it for :)

I have no problem with changing its semantics later on if you want to
build on top of it, rename it, anything - so far as the LRU balancing
is unaffected of course.

But I don't think it makes sense to provision it for potential future
cases that may or may not materialize.

> > > Do we want to retain [1]?
> > > 
> > > This patch motivates from swap IO could be much faster than file IO
> > > so that it would be natural if we rely on refaulting feedback rather
> > > than forcing evicting file cache?
> > > 
> > > [1] e9868505987a, mm,vmscan: only evict file pages when we have plenty?
> > 
> > Yes! We don't want to go after the workingset, whether it be cache or
> > anonymous, while there is single-use page cache lying around that we
> > can reclaim for free, with no IO and little risk of future IO. Anon
> > memory doesn't have this equivalent. Only cache is lazy-reclaimed.
> > 
> > Once the cache refaults, we activate it to reflect the fact that it's
> > workingset. Only when we run out of single-use cache do we want to
> > reclaim multi-use pages, and *then* we balance workingsets based on
> > cost of refetching each side from secondary storage.
> 
> If pages in inactive file LRU are really single-use page cache, I agree.
> 
> However, how does the logic can work like that?
> If reclaimed file pages were part of workingset(i.e., refault happens),
> we give the pressure to anonymous LRU but get_scan_count still force to
> reclaim file lru until inactive file LRU list size is enough low.
> 
> With that, too many file workingset could be evicted although anon swap
> is cheaper on fast swap storage?
> 
> IOW, refault mechanisme works once inactive file LRU list size is enough
> small but small inactive file LRU doesn't guarantee it has only multiple
> -use pages. Hm, Isn't it a problem?

It's a trade-off between the cost of detecting a new workingset from a
stream of use-once pages, and the cost of use-once pages impose on the
established workingset.

That's a pretty easy choice, if you ask me. I'd rather ask cache pages
to prove they are multi-use than have use-once pages put pressure on
the workingset.

Sure, a spike like you describe is certainly possible, where a good
portion of the inactive file pages will be re-used in the near future,
yet we evict all of them in a burst of memory pressure when we should
have swapped. That's a worst case scenario for the use-once policy in
a workingset transition.

However, that's much better than use-once pages, which cost no
additional IO to reclaim and do not benefit from being cached at all,
causing the workingset to be trashed or swapped out.

In your scenario, the real multi-use pages will quickly refault and
get activated and the algorithm will adapt to the new circumstances.