Re: [PATCH -V10 RESEND 0/6] NUMA balancing: optimize memory placement for memory tiering system

[Huang, Ying]

"Huang, Ying" <ying.huang@xxxxxxxxx> writes:

> Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
>
>> On Thu, Jan 13, 2022 at 08:06:40PM +0800, Huang, Ying wrote:
>>> Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
>>> > On Thu, Jan 13, 2022 at 03:19:06PM +0800, Huang, Ying wrote:
>>> >> Peter Zijlstra <peterz@xxxxxxxxxxxxx> writes:
>>> >> > On Tue, Dec 07, 2021 at 10:27:51AM +0800, Huang Ying wrote:
>>
>>> >> >> After commit c221c0b0308f ("device-dax: "Hotplug" persistent memory
>>> >> >> for use like normal RAM"), the PMEM could be used as the
>>> >> >> cost-effective volatile memory in separate NUMA nodes.  In a typical
>>> >> >> memory tiering system, there are CPUs, DRAM and PMEM in each physical
>>> >> >> NUMA node.  The CPUs and the DRAM will be put in one logical node,
>>> >> >> while the PMEM will be put in another (faked) logical node.
>>> >> >
>>> >> > So what does a system like that actually look like, SLIT table wise, and
>>> >> > how does that affect init_numa_topology_type() ?
>>> >> 
>>> >> The SLIT table is as follows,
>>
>> <snip>
>>
>>> >> node distances:
>>> >> node   0   1   2   3 
>>> >>   0:  10  21  17  28 
>>> >>   1:  21  10  28  17 
>>> >>   2:  17  28  10  28 
>>> >>   3:  28  17  28  10 
>>> >> 
>>> >> init_numa_topology_type() set sched_numa_topology_type to NUMA_DIRECT.
>>> >> 
>>> >> The node 0 and node 1 are onlined during boot.  While the PMEM node,
>>> >> that is, node 2 and node 3 are onlined later.  As in the following dmesg
>>> >> snippet.
>>> >
>>> > But how? sched_init_numa() scans the *whole* SLIT table to determine
>>> > nr_levels / sched_domains_numa_levels, even offline nodes. Therefore it
>>> > should find 4 distinct distance values and end up not selecting
>>> > NUMA_DIRECT.
>>> >
>>> > Similarly for the other types it uses for_each_online_node(), which
>>> > would include the pmem nodes once they've been onlined, but I'm thinking
>>> > we explicitly want to skip CPU-less nodes in that iteration.
>>> 
>>> I used the debug patch as below, and get the log in dmesg as follows,
>>> 
>>> [    5.394577][    T1] sched_numa_topology_type: 0, levels: 4, max_distance: 28
>>> 
>>> I found that I forget another caller of init_numa_topology_type() run
>>> during hotplug.  I will add another printk() to show it.  Sorry about
>>> that.
>>
>> Can you try with this on?
>>
>> I'm suspecting there's a problem with init_numa_topology_type(); it will
>> never find the max distance due to the _online_ clause in the iteration,
>> since you said the pmem nodes are not online yet.
>>
>> ---
>> diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
>> index d201a7052a29..53ab9c63c185 100644
>> --- a/kernel/sched/topology.c
>> +++ b/kernel/sched/topology.c
>> @@ -1756,6 +1756,8 @@ static void init_numa_topology_type(void)
>>  			return;
>>  		}
>>  	}
>> +
>> +	WARN(1, "no NUMA type determined");
>>  }
>
> Hi, Peter,
>
> I have run the test, the warning is triggered in the dmesg as follows.
> I will continue to debug hotplug tomorrow.

I did more experiments and found that init_numa_topology_type() will not
be called during PMEM nodes plugging in.  Because it will only be called
when a CPU of a never-onlined-before node is onlined.  There's no CPU on
the PMEM nodes (2/3).  So, when the PMEM node is onlined,
init_numa_topology_type() will not be called. And
sched_numa_topology_type will not be changed.

Best Regards,
Huang, Ying