Re: RFC: Memory Tiering Kernel Interfaces (v2)

[Wei Xu]

On Fri, May 13, 2022 at 12:04 AM ying.huang@xxxxxxxxx
<ying.huang@xxxxxxxxx> wrote:
>
> On Thu, 2022-05-12 at 23:36 -0700, Wei Xu wrote:
> > On Thu, May 12, 2022 at 8:25 PM ying.huang@xxxxxxxxx
> > <ying.huang@xxxxxxxxx> wrote:
> > >
> > > On Wed, 2022-05-11 at 23:22 -0700, Wei Xu wrote:
> > > >
> > > > Memory Allocation for Demotion
> > > > ==============================
> > > >
> > > > To allocate a new page as the demotion target for a page, the kernel
> > > > calls the allocation function (__alloc_pages_nodemask) with the
> > > > source page node as the preferred node and the union of all lower
> > > > tier nodes as the allowed nodemask.  The actual target node selection
> > > > then follows the allocation fallback order that the kernel has
> > > > already defined.
> > > >
> > > > The pseudo code looks like:
> > > >
> > > >     targets = NODE_MASK_NONE;
> > > >     src_nid = page_to_nid(page);
> > > >     src_tier = node_tier_map[src_nid];
> > > >     for (i = src_tier + 1; i < MAX_MEMORY_TIERS; i++)
> > > >             nodes_or(targets, targets, memory_tiers[i]);
> > > >     new_page = __alloc_pages_nodemask(gfp, order, src_nid, targets);
> > > >
> > > > The memopolicy of cpuset, vma and owner task of the source page can
> > > > be set to refine the demotion target nodemask, e.g. to prevent
> > > > demotion or select a particular allowed node as the demotion target.
> > >
> > > Consider a system with 3 tiers, if we want to demote some pages from
> > > tier 0, the desired behavior is,
> > >
> > > - Allocate pages from tier 1
> > > - If there's no enough free pages in tier 1, wakeup kswapd of tier 1 so
> > > demote some pages from tier 1 to tier 2
> > > - If there's still no enough free pages in tier 1, allocate pages from
> > > tier 2.
> > >
> > > In this way, tier 0 will have the hottest pages, while tier 1 will have
> > > the coldest pages.
> >
> > When we are already in the allocation path for the demotion of a page
> > from tier 0, I think we'd better not block this allocation to wait for
> > kswapd to demote pages from tier 1 to tier 2. Instead, we should
> > directly allocate from tier 2.  Meanwhile, this demotion can wakeup
> > kswapd to demote from tier 1 to tier 2 in the background.
>
> Yes.  That's what I want too.  My original words may be misleading.
>
> > > With your proposed method, the demoting from tier 0 behavior is,
> > >
> > > - Allocate pages from tier 1
> > > - If there's no enough free pages in tier 1, allocate pages in tier 2
> > >
> > > The kswapd of tier 1 will not be waken up until there's no enough free
> > > pages in tier 2.  In quite long time, there's no much hot/cold
> > > differentiation between tier 1 and tier 2.
> >
> > This is true with the current allocation code. But I think we can make
> > some changes for demotion allocations. For example, we can add a
> > GFP_DEMOTE flag and update the allocation function to wake up kswapd
> > when this flag is set and we need to fall back to another node.
> >
> > > This isn't hard to be fixed, just call __alloc_pages_nodemask() for each
> > > tier one by one considering page allocation fallback order.
> >
> > That would have worked, except that there is an example earlier, in
> > which it is actually preferred for some nodes to demote to their tier
> > + 2, not tier +1.
> >
> > More specifically, the example is:
> >
> >                  20
> >    Node 0 (DRAM) -- Node 1 (DRAM)
> >     |   |           |    |
> >     |   | 30    120 |    |
> >     |   v           v    | 100
> > 100 |  Node 2 (PMEM)     |
> >     |    |               |
> >     |    | 100           |
> >      \   v               v
> >       -> Node 3 (Large Mem)
> >
> > Node distances:
> > node   0    1    2    3
> >    0  10   20   30  100
> >    1  20   10  120  100
> >    2  30  120   10  100
> >    3 100  100  100   10
> >
> > 3 memory tiers are defined:
> > tier 0: 0-1
> > tier 1: 2
> > tier 2: 3
> >
> > The demotion fallback order is:
> > node 0: 2, 3
> > node 1: 3, 2
> > node 2: 3
> > node 3: empty
> >
> > Note that even though node 3 is in tier 2 and node 2 is in tier 1,
> > node 1 (tier 0) still prefers node 3 as its first demotion target, not
> > node 2.
>
> Yes.  I understand that we need to support this use case.  We can use
> the tier order in allocation fallback list instead of from small to
> large.  That is, for node 1, the tier order for demotion is tier 2, tier
> 1.

That could work, too, though I feel it might be simpler and more
efficient (no repeated calls to __alloc_pages for the same allocation)
to modify __alloc_pages() itself.

Anyway, we can discuss more on this when it comes to the
implementation of this demotion allocation function.  I believe this
should not affect the general memory tiering interfaces proposed here.

> Best Regards,
> Huang, Ying
>
>
>