Re: [PATCH -V10 0/9] Migrate Pages in lieu of discard

[Dave Hansen]

On 7/15/21 12:38 PM, Andrew Morton wrote:
> On Thu, 15 Jul 2021 13:51:36 +0800 Huang Ying <ying.huang@xxxxxxxxx> wrote:
>> We have tested the patchset with the postgresql and pgbench.  On a
>> 2-socket server machine with DRAM and PMEM, the kernel with the
>> patchset can improve the score of pgbench up to 22.1% compared with
>> that of the DRAM only + disk case.  This comes from the reduced disk
>> read throughput (which reduces up to 70.8%).
> 
> The [0/n] description talks a lot about PMEM, but the patches
> themselves are all about NUMA nodes.  I assume that what ties this
> together is that the PMEM tends to be organized as a NUMA node on its
> own, and that by enabling migrate-to-remote-node-during-reclaim, we get
> this PMEM behaviour as a desired side-effect?

Yes, an assumption of this whole thing is that there are uniform
performance capabilities within a NUMA node.  ACPI systems that
implement the HMAT table require that each proximity domain (PXM) have
these uniform capabilities.  This works out great since PXMs are so
closely translated to NUMA nodes.

> IOW, perhaps this [0/n] description could explain the linkage between
> PMEM and NUMA nodes more explicitly.

Sure, we can do that.

> Secondly...  at some point it just won't make sense to migrate pages to
> a remote node because that remote node isn't remote enough.  I mean, if
> accessing the remote node is 2% slower than the local node, it's best
> to just reclaim the page rather than going to the effort of migrating
> it around?

One thing to keep in mind is that these migrations are *cheap*.  Because
of the point where the migration is done near the end of the reclaim
process the page should be unmapped.  The expensive part like TLB
shootdowns don't need to happen.

But, yes, you're right.  There are going to be situations where it would
have been better to reclaim the page.

> And I assume the patchset doesn't (yet) attempt to make this judgement?

Correct.  The assumption is that, on average, having something in slower
memory is better than having it out of memory.

> Thirdly, the final patch which makes the feature off-by-default is a
> concern.  I expect this will cause the new feature to have much less
> testing (functional and performance).  We could make it default-on for
> now, then flip that to default-off after 5.14-rc5 (for example).  That
> will help, but is a bit lame.  Is it not possible for the kernel to
> look at the overall system layout and make an educated guess as to
> whether this system will benefit?

Unfortunately, no.  It's highly workload dependent.  I see it a lot like
autonuma or khugepaged/THP: there are going to be some big upsides, but
we're under no illusions that the wins will be universal.

I'd actually be quite happy to flip it to be default-on all the time.
But, I think I'll be a lot more confident about it once we have the
"promotion" side of the equation merged as well:

	https://lwn.net/Articles/835402/

> And I see this, from the [9/9] changelog:
> 
> : The benefits are truly workload-based and *not* hardware-based.  We do
> : not believe that there is a viable threshold where certain hardware
> : configurations should have this mechanism enabled while others do not.
> 
> so...  why not simply enable the thing for all systems and be done with
> it?  What sort of downsides are you anticipating?  We could of course
> include an emergency-off knob, but hoping that it won't need to be
> used.

The worst case scenario is that an important workload starts up, eats a
bunch of RAM and then goes idle, maybe during off-peak workload hours.
During the night, a bunch of batch jobs kick in and evict some of that
workload's data to PMEM.  The workload spools up again the next morning
and is hampered by the slower PMEM when there's lots of DRAM available.

This will be helped by the autonuma-based promotion once it's in place.

> Finally, having a feature which is good for some workloads, bad for
> others and which contains a system-wide enable knob is really quite
> sad.  It requires a lot of work from each and every operator in the
> world!  They must experimentally run their workloads both with and
> without, and determine which is best.  And they should rerun this
> testing periodically as userspace and the kernel evolve, to determine
> whether their earlier experimenting holds true.  And what happens if
> workload #1 wins and workload #2 loses?
> 
> And of course, many operators simply won't do all of this and they'll
> run slower, or will miss out of benefits.  What can we do to relieve
> our users of all of this?

I think this is a much bigger question than this patch set.  It's
basically the tale of /proc/sys/vm/* or /sys/kernel/mm.  Some workloads
win with transparent_hugepage/enabled=always, some lose.  Some win with
zone_reclaim_mode=7, some prefer zone_reclaim_mode=0.

The end game for auto-migration is to on by default, wherever these
hardware configurations show up.  Just like THP, I expect this to be
broadly useful.  But, given the breadth of our user base, I also expect
it to be nasty for a minority of users.  They'll need a way to turn it
off, and hopefully they'll also provide feedback so that, if possible,
we can improve the default behavior and bring them back into the fold.