Re: [PATCH v8 00/12] mm/demotion: Memory tiers and demotion

[Huang, Ying]

Aneesh Kumar K V <aneesh.kumar@xxxxxxxxxxxxx> writes:

> On 7/14/22 10:26 AM, Huang, Ying wrote:
>> "Aneesh Kumar K.V" <aneesh.kumar@xxxxxxxxxxxxx> writes:
>> 
>>> "Huang, Ying" <ying.huang@xxxxxxxxx> writes:
>> 
>> [snip]
>> 
>>>>
>>>> I believe that sparse memory tier IDs can make memory tier more stable
>>>> in some cases.  But this is different from the system suggested by
>>>> Johannes.  Per my understanding, with Johannes' system, we will
>>>>
>>>> - one driver may online different memory types (such as kmem_dax may
>>>>   online HBM, PMEM, etc.)
>>>>
>>>> - one memory type manages several memory nodes (NUMA nodes)
>>>>
>>>> - one "abstract distance" for each memory type
>>>>
>>>> - the "abstract distance" can be offset by user space override knob
>>>>
>>>> - memory tiers generated dynamic from different memory types according
>>>>   "abstract distance" and overridden "offset"
>>>>
>>>> - the granularity to group several memory types into one memory tier can
>>>>   be overridden via user space knob
>>>>
>>>> In this way, the memory tiers may be changed totally after user space
>>>> overridden.  It may be hard to link memory tiers before/after the
>>>> overridden.  So we may need to reset all per-memory-tier configuration,
>>>> such as cgroup paritation limit or interleave weight, etc.
>>>
>>> Making sure we all agree on the details.
>>>
>>> In the proposal https://lore.kernel.org/linux-mm/7b72ccf4-f4ae-cb4e-f411-74d055482026@xxxxxxxxxxxxx
>>> instead of calling it "abstract distance" I was referring it as device
>>> attributes.
>>>
>>> Johannes also suggested these device attributes/"abstract distance"
>>> to be used to derive the memory tier to which the memory type/memory
>>> device will be assigned.
>>>
>>> So dax kmem would manage different types of memory and based on the device
>>> attributes, we would assign them to different memory tiers (memory tiers
>>> in the range [0-200)).
>>>
>>> Now the additional detail here is that we might add knobs that will be
>>> used by dax kmem to fine-tune memory types to memory tiers assignment.
>>> On updating these knob values, the kernel should rebuild the entire
>>> memory tier hierarchy. (earlier I was considering only newly added
>>> memory devices will get impacted by such a change. But I agree it
>>> makes sense to rebuild the entire hierarchy again) But that rebuilding
>>> will be restricted to dax kmem driver.
>>>
>> 
>> Thanks for explanation and pointer.  Per my understanding, memory
>> types and memory devices including abstract distances are used to
>> describe the *physical* memory devices, not *policy*.  We may add more
>> physical attributes to these memory devices, such as, latency,
>> throughput, etc.  I think we can reach consensus on this point?
>> 
>> In contrast, memory tiers are more about policy, such as
>> demotion/promotion, interleaving and possible partition among cgroups.
>> How to derive memory tiers from memory types (or devices)?  We have
>> multiple choices.
>> 
>
> agreed to the above.
>
>> Per my understanding, Johannes suggested to use some policy parameters
>> such as distance granularity (e.g., if granularity is 100, then memory
>> devices with abstract distance 0-100, 100-200, 200-300, ... will be put
>> to memory tier 0, 1, 2, ...) to build the memory tiers.  Distance
>> granularity may be not flexible enough, we may need something like a set
>> of cutoffs or range, e.g., 50, 100, 200, 500, or 0-50, 50-100, 100-200,
>> 200-500, >500.  These policy parameters should be overridable from user
>> space.
>> 
>
> The term distance was always confusing to me. Instead, I was
> generalizing it as an attribute.

Attributes sounds too general to me :-)

> The challenge with the term distance for me was in clarifying the
> distance of this memory device from where? Instead, it is much simpler
> to group devices based on device attributes such as write latency.

Per my understanding, the "distance" here is the distance from local
CPUs, that is, get rid of the influence of NUMA topology as much as
possible.

There may be other memory accessing initiators in the system, such as
GPU, etc.  But we don't want to have different memory tiers for each
initiators, so we mainly consider CPUs.  The device drivers of other
initiators may consider other type of memory tiers.

The "distance" characters the latency of the memory device under typical
memory throughput in the system.  So it characterizes both latency and
throughput, because the latency will increase with the throughput.  This
one of reasons we need to override the default distance, because the
typical memory throughput may be different among different workloads.

The "abstract distance" can come from SLIT, HMAT firstly.  Then we can
try to explore the other possible sources of information.

> So everything you explained above is correct, except we describe it in terms of a
> single device attribute or a combination of multiple device attributes. We could convert
> a combination of multiple device attribute to an "abstract distance".

Sounds good to me.

> Such an "abstract distance" is derived based on different device
> attribute values with policy parameters overridable from userspace.

I think "abstract distance" is different from policy parameters.

>> And per my understanding, you suggested to place memory devices to
>> memory tiers directly via a knob of memory types (or memory devices).
>> e.g., memory_type/memtier can be written to place the memory devices of
>> the memory_type to the specified memtier.  Or via
>> memorty_type/distance_offset to do that.
>> 
>
> What I explained above is what I would expect the kernel to do by default. Before we can
> reach there we need to get a better understanding of which device attribute describes
> the grouping of memory devices to a memory tier. Do we need latency-based grouping
> or bandwidth-based grouping? Till then userspace can place these devices to different
> memory tiers. Hence the addition of /sys/devices/system/node/nodeN/memtier write feature
> which moves a memory node to a specific memory tier. 
>
> I am not suggesting we override the memory types from userspace.

OK.  I don't think we need this.  We can examine the target solution
above and try to find any issue with it.

Best Regards,
Huang, Ying