Re: [PATCH RESEND 4/4] dax, kmem: calculate abstract distance with general interface

[Huang, Ying]

Alistair Popple <apopple@xxxxxxxxxx> writes:

> "Huang, Ying" <ying.huang@xxxxxxxxx> writes:
>
>> Alistair Popple <apopple@xxxxxxxxxx> writes:
>>
>>> "Huang, Ying" <ying.huang@xxxxxxxxx> writes:
>>>
>>>> Alistair Popple <apopple@xxxxxxxxxx> writes:
>>>>
>>>>> Huang Ying <ying.huang@xxxxxxxxx> writes:
>>>>>
>>>>>> Previously, a fixed abstract distance MEMTIER_DEFAULT_DAX_ADISTANCE is
>>>>>> used for slow memory type in kmem driver.  This limits the usage of
>>>>>> kmem driver, for example, it cannot be used for HBM (high bandwidth
>>>>>> memory).
>>>>>>
>>>>>> So, we use the general abstract distance calculation mechanism in kmem
>>>>>> drivers to get more accurate abstract distance on systems with proper
>>>>>> support.  The original MEMTIER_DEFAULT_DAX_ADISTANCE is used as
>>>>>> fallback only.
>>>>>>
>>>>>> Now, multiple memory types may be managed by kmem.  These memory types
>>>>>> are put into the "kmem_memory_types" list and protected by
>>>>>> kmem_memory_type_lock.
>>>>>
>>>>> See below but I wonder if kmem_memory_types could be a common helper
>>>>> rather than kdax specific?
>>>>>
>>>>>> Signed-off-by: "Huang, Ying" <ying.huang@xxxxxxxxx>
>>>>>> Cc: Aneesh Kumar K.V <aneesh.kumar@xxxxxxxxxxxxx>
>>>>>> Cc: Wei Xu <weixugc@xxxxxxxxxx>
>>>>>> Cc: Alistair Popple <apopple@xxxxxxxxxx>
>>>>>> Cc: Dan Williams <dan.j.williams@xxxxxxxxx>
>>>>>> Cc: Dave Hansen <dave.hansen@xxxxxxxxx>
>>>>>> Cc: Davidlohr Bueso <dave@xxxxxxxxxxxx>
>>>>>> Cc: Johannes Weiner <hannes@xxxxxxxxxxx>
>>>>>> Cc: Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx>
>>>>>> Cc: Michal Hocko <mhocko@xxxxxxxxxx>
>>>>>> Cc: Yang Shi <shy828301@xxxxxxxxx>
>>>>>> Cc: Rafael J Wysocki <rafael.j.wysocki@xxxxxxxxx>
>>>>>> ---
>>>>>>  drivers/dax/kmem.c           | 54 +++++++++++++++++++++++++++---------
>>>>>>  include/linux/memory-tiers.h |  2 ++
>>>>>>  mm/memory-tiers.c            |  2 +-
>>>>>>  3 files changed, 44 insertions(+), 14 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
>>>>>> index 898ca9505754..837165037231 100644
>>>>>> --- a/drivers/dax/kmem.c
>>>>>> +++ b/drivers/dax/kmem.c
>>>>>> @@ -49,14 +49,40 @@ struct dax_kmem_data {
>>>>>>  	struct resource *res[];
>>>>>>  };
>>>>>>  
>>>>>> -static struct memory_dev_type *dax_slowmem_type;
>>>>>> +static DEFINE_MUTEX(kmem_memory_type_lock);
>>>>>> +static LIST_HEAD(kmem_memory_types);
>>>>>> +
>>>>>> +static struct memory_dev_type *kmem_find_alloc_memorty_type(int adist)
>>>>>> +{
>>>>>> +	bool found = false;
>>>>>> +	struct memory_dev_type *mtype;
>>>>>> +
>>>>>> +	mutex_lock(&kmem_memory_type_lock);
>>>>>> +	list_for_each_entry(mtype, &kmem_memory_types, list) {
>>>>>> +		if (mtype->adistance == adist) {
>>>>>> +			found = true;
>>>>>> +			break;
>>>>>> +		}
>>>>>> +	}
>>>>>> +	if (!found) {
>>>>>> +		mtype = alloc_memory_type(adist);
>>>>>> +		if (!IS_ERR(mtype))
>>>>>> +			list_add(&mtype->list, &kmem_memory_types);
>>>>>> +	}
>>>>>> +	mutex_unlock(&kmem_memory_type_lock);
>>>>>> +
>>>>>> +	return mtype;
>>>>>> +}
>>>>>> +
>>>>>>  static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>>>>>>  {
>>>>>>  	struct device *dev = &dev_dax->dev;
>>>>>>  	unsigned long total_len = 0;
>>>>>>  	struct dax_kmem_data *data;
>>>>>> +	struct memory_dev_type *mtype;
>>>>>>  	int i, rc, mapped = 0;
>>>>>>  	int numa_node;
>>>>>> +	int adist = MEMTIER_DEFAULT_DAX_ADISTANCE;
>>>>>>  
>>>>>>  	/*
>>>>>>  	 * Ensure good NUMA information for the persistent memory.
>>>>>> @@ -71,6 +97,11 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>>>>>>  		return -EINVAL;
>>>>>>  	}
>>>>>>  
>>>>>> +	mt_calc_adistance(numa_node, &adist);
>>>>>> +	mtype = kmem_find_alloc_memorty_type(adist);
>>>>>> +	if (IS_ERR(mtype))
>>>>>> +		return PTR_ERR(mtype);
>>>>>> +
>>>>>
>>>>> I wrote my own quick and dirty module to test this and wrote basically
>>>>> the same code sequence.
>>>>>
>>>>> I notice your using a list of memory types here though. I think it would
>>>>> be nice to have a common helper that other users could call to do the
>>>>> mt_calc_adistance() / kmem_find_alloc_memory_type() /
>>>>> init_node_memory_type() sequence and cleanup as my naive approach would
>>>>> result in a new memory_dev_type per device even though adist might be
>>>>> the same. A common helper would make it easy to de-dup those.
>>>>
>>>> If it's useful, we can move kmem_find_alloc_memory_type() to
>>>> memory-tier.c after some revision.  But I tend to move it after we have
>>>> the second user.  What do you think about that?
>>>
>>> Usually I would agree, but this series already introduces a general
>>> interface for calculating adist even though there's only one user and
>>> implementation. So if we're going to add a general interface I think it
>>> would be better to make it more usable now rather than after variations
>>> of it have been cut and pasted into other drivers.
>>
>> In general, I would like to introduce complexity when necessary.  So, we
>> can discuss the necessity of the general interface firstly.  We can do
>> that in [1/4] of the series.
>
> Do we need one memory_dev_type per adistance or per adistance+device?
>
> If IUC correctly I think it's the former. Logically that means
> memory_dev_types should be managed by the memory-tiering subsystem
> because they are system wide rather than driver specific resources. That
> we need to add the list field to struct memory_dev_type specifically for
> use by dax/kmem supports that idea.

In the original design (page 9/10/11 of [1]), memory_dev_type (Memory
Type) is driver specific.

[1] https://lpc.events/event/16/contributions/1209/attachments/1042/1995/Live%20In%20a%20World%20With%20Multiple%20Memory%20Types.pdf

Memory devices with same adistance but was enumerated by different
drivers will be put in different memory_dev_type.  As a not-so-realistic
example, even if the PMEM devices in DIMM slots and the CXL.mem devices
in CXL slots have same adistance, their memory_dev_type will be
different.  Because the memory devices enumerated by the same driver
should be managed in the same way, but memory devices enumerated by
different drivers may be not.  For example, if we adjust the adistance
of the PMEM devices to put them in the lower memory tier via
not-yet-implemented abstract_distance_offset sysfs knob, we may not want
to adjust it of CXL.mem devices at the same time.

> Also I'm not sure why you consider moving the
> kmem_memory_types/kmem_find_alloc_memory_type()/etc. functions into
> mm/memory-tiers.c to add complexity. Isn't it just moving code around or
> am I missing some other subtlety that makes this hard? I really think
> logically memory-tiering.c is where management of the various
> memory_dev_types belongs.

IMHO, it depends on whether these functions are shared by at least 2
drivers.  If so, we can put them in mm/memory-tiers.c.  Otherwise, we
should keep them in the driver.

--
Best Regards,
Huang, Ying