Re: [PATCH v3 updated] mm/demotion: Expose memory tier details via sysfs

[Aneesh Kumar K V]

On 9/2/22 1:27 PM, Huang, Ying wrote:
> Wei Xu <weixugc@xxxxxxxxxx> writes:
> 
>> On Thu, Sep 1, 2022 at 11:44 PM Aneesh Kumar K V
>> <aneesh.kumar@xxxxxxxxxxxxx> wrote:
>>>
>>> On 9/2/22 12:10 PM, Huang, Ying wrote:
>>>> Aneesh Kumar K V <aneesh.kumar@xxxxxxxxxxxxx> writes:
>>>>
>>>>> On 9/2/22 11:42 AM, Huang, Ying wrote:
>>>>>> Aneesh Kumar K V <aneesh.kumar@xxxxxxxxxxxxx> writes:
>>>>>>
>>>>>>> On 9/2/22 11:10 AM, Huang, Ying wrote:
>>>>>>>> Aneesh Kumar K V <aneesh.kumar@xxxxxxxxxxxxx> writes:
>>>>>>>>
>>>>>>>>> On 9/2/22 10:39 AM, Wei Xu wrote:
>>>>>>>>>> On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying <ying.huang@xxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Aneesh Kumar K V <aneesh.kumar@xxxxxxxxxxxxx> writes:
>>>>>>>>>>>
>>>>>>>>>>>> On 9/1/22 12:31 PM, Huang, Ying wrote:
>>>>>>>>>>>>> "Aneesh Kumar K.V" <aneesh.kumar@xxxxxxxxxxxxx> writes:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier
>>>>>>>>>>>>>> related details can be found. All allocated memory tiers will be listed
>>>>>>>>>>>>>> there as /sys/devices/virtual/memory_tiering/memory_tierN/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The nodes which are part of a specific memory tier can be listed via
>>>>>>>>>>>>>> /sys/devices/virtual/memory_tiering/memory_tierN/nodes
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think "memory_tier" is a better subsystem/bus name than
>>>>>>>>>>>>> memory_tiering.  Because we have a set of memory_tierN devices inside.
>>>>>>>>>>>>> "memory_tier" sounds more natural.  I know this is subjective, just my
>>>>>>>>>>>>> preference.
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4
>>>>>>>>> because we would want it to a susbsystem where all memory tiering related details can be found
>>>>>>>>> including memory type in the future. This is as per discussion
>>>>>>>>>
>>>>>>>>> https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@xxxxxxxxxxxxxx
>>>>>>>>
>>>>>>>> I don't think that it's a good idea to mix 2 types of devices in one
>>>>>>>> subsystem (bus).  If my understanding were correct, that breaks the
>>>>>>>> driver core convention.
>>>>>>>>
>>>>>>>
>>>>>>> All these are virtual devices .I am not sure i follow what you mean by 2 types of devices.
>>>>>>> memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows
>>>>>>> details of memory tiers and can possibly contain details of different memory types .
>>>>>>
>>>>>> IMHO, memory_tier and memory_type are 2 kind of devices.  They have
>>>>>> almost totally different attributes (sysfs file).  So, we should create
>>>>>> 2 buses for them.  Each has its own attribute group.  "virtual" itself
>>>>>> isn't a subsystem.
>>>>>
>>>>> Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate
>>>>> them within the same subdirectory? I am still not clear why you are suggesting they need to be in different
>>>>> sysfs hierarchy.  It doesn't break any driver core convention as you mentioned earlier.
>>>>>
>>>>> /sys/devices/virtual/memory_tiering/memory_tierN
>>>>> /sys/devices/virtual/memory_tiering/memory_typeN
>>>>
>>>> I think we should add
>>>>
>>>>  /sys/devices/virtual/memory_tier/memory_tierN
>>>>  /sys/devices/virtual/memory_type/memory_typeN
>>>>
>>>
>>> I am trying to find if there is a technical reason to do the same?
>>>
>>>> I don't think this is complex.  Devices of same bus/subsystem should
>>>> have mostly same attributes.  This is my understanding of driver core
>>>> convention.
>>>>
>>>
>>> I was not looking at this from code complexity point. Instead of having multiple directories
>>> with details w.r.t memory tiering, I was looking at consolidating the details
>>> within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices
>>> are consolidated within /sys/devics/virtual/).
>>>
>>> -aneesh
>>
>> Here is an example of /sys/bus/nd/devices (I know it is not under
>> /sys/devices/virtual, but it can still serve as a reference):
>>
>> ls -1 /sys/bus/nd/devices
>>
>> namespace2.0
>> namespace3.0
>> ndbus0
>> nmem0
>> nmem1
>> region0
>> region1
>> region2
>> region3
>>
>> So I think it is not unreasonable if we want to group memory tiering
>> related interfaces within a single top directory.
> 
> Thanks for pointing this out.  My original understanding of driver core
> isn't correct.
> 
> But I still think it's better to separate instead of mixing memory_tier
> and memory_type.  Per my understanding, memory_type shows information
> (abstract distance, latency, bandwidth, etc.) of memory types (and
> nodes), it can be useful even without memory tiers.  That is, memory
> types describes the physical characteristics, while memory tier reflects
> the policy.
>

The latency and bandwidth details are already exposed via 

	/sys/devices/system/node/nodeY/access0/initiators/

Documentation/admin-guide/mm/numaperf.rst

That is the interface that libraries like libmemkind will look at for finding
details w.r.t latency/bandwidth

-aneesh