Re: Ways to deprecate /sys/devices/system/memory/memoryX/phys_device ?

[David Hildenbrand]

On 11.09.20 11:12, Michal Hocko wrote:
> On Fri 11-09-20 10:09:07, David Hildenbrand wrote:
> [...]
>> Consider two cases:
>>
>> 1. Hot(un)plugging huge DIMMs: many (not all!) use cases want to
>> online/offline the whole thing. HW can effectively only plug/unplug the
>> whole thing. It makes sense in some (most?) setups to represent one DIMM
>> as one memory block device.
> 
> Yes, for the physical hotplug it doesn't really make much sense to me to
> offline portions that the HW cannot hotremove.

I've seen people offline parts of memory to simulate systems with less
RAM and people offline parts of memory on demand to save energy
(poweroff banks). People won't stop being creative with what we provided
to them :D

> 
>> 2. Hot(un)plugging small memory increments. This is mostly the case in
>> virtualized environments - especially hyper-v balloon, xen balloon,
>> virtio-mem and (drumroll) ppc dlpar and s390x standby memory. On PPC,
>> you want at least all (16MB!) memory block devices that can get
>> unplugged again individually ("LMBs") as separate memory blocks. Same on
>> s390x on memory increment size (currently effectively the memory block
>> size).
> 
> Yes I do recognize those usecase even though I will not pretend I
> consider it quesitonable. E.g. any hotplug with a smaller granularity
> than the memory model in Linus allows is just dubious. We simply cannot
> implement that without a lot of wasting and then the question is what is
> the real point.

Having the section size as small as possible in these environments is
most certainly preferable, to clean up metadata where possible.
Otherwise, hot(un)plugging smaller granularity behaves more like memory
ballooning (and I think I don't have to tell you that ballooning is used
excessively even though it wastes memory on metadata ;) ). Anyhow,
that's another discussion.

> 
>> In summary, larger memory block devices mostly only make sense with
>> DIMMs (and for boot memory in some cases). We will still end up with
>> many memory block devices in other configurations.
> 
> And that is fine because the boot time memory is still likely the
> primary source of memory. And reducing memory devices for those is a
> huge improvement already (just think of a multi TB system with
> gazillions pointless memory devices). 

Agreed. On my workstation (64GB - 4x16GB DIMMs if I recall correctly) I
end up with

$ cat /sys/devices/system/memory/block_size_bytes
8000000
$ ls /sys/devices/system/memory/ | grep memory | wc -l
512

$ cat /proc/iomem
00000000-00000fff : Reserved
00001000-0009ffff : System RAM
000a0000-000fffff : Reserved
  000a0000-000bffff : PCI Bus 0000:00
  000c0000-000dffff : PCI Bus 0000:00
    000c0000-000cf1ff : Video ROM
  000f0000-000fffff : System ROM
00100000-09dfffff : System RAM
09e00000-09ffffff : Reserved
0a000000-0a1fffff : System RAM
0a200000-0a20ffff : ACPI Non-volatile Storage
0a210000-b70fe017 : System RAM
b70fe018-b7117c57 : System RAM
b7117c58-b7118017 : System RAM
b7118018-b7129057 : System RAM
b7129058-b826cfff : System RAM
b826d000-b82c3fff : Reserved
b82c4000-b8d52fff : System RAM
b8d53000-b8d53fff : Reserved
b8d54000-bc67cfff : System RAM
bc67d000-bca26fff : Reserved
bca27000-bca73fff : ACPI Tables
bca74000-bd103fff : ACPI Non-volatile Storage
bd104000-bddfefff : Reserved
bddff000-beffffff : System RAM
bf000000-bfffffff : Reserved
[ PCI stuff ]
100000000-103f2fffff : System RAM
  d9f000000-d9fe00d90 : Kernel code
  da0000000-da07f9fff : Kernel rodata
  da0800000-da0a59e3f : Kernel data
  da110c000-da15fffff : Kernel bss
103f300000-10503fffff : Reserved


If we'd want to create a separate device during boot for each "System
RAM" resource, I am having a hard time figuring out the actual devices
(4 DIMMs). For memory hotplug it's a lot easier (e.g., separate
add_memory() calls). Of course, my workstation most probably doesn't
support DIMM hot(un)plug, so the BIOS might do strange things.

Also, I do wonder how hard the BIOS might mess up a DIMM configuration
(e820 map, resulting in "System RAM" resources) after hotplug, when
rebooting - or after kexec.

On bare metal, people expect that DIMMs that where hotplugged can be
hotunplugged again after reboot (of course, taking care of ZONE_MOVABLE,
which is a pain). As discussed under QEMU that's easier, because we get
separate add_memory() calls for all DIMMs from ACPI code. How stuff
behaves on bare metal is still a head-scratcher -  if we can rely on
separate "System RAM" instances to cover separate DIMMs, or if DIMMs
might get merged/split/EFI allocations ...

Maybe we can derive the actual DIMMs from some ACPI tables (SRAT?),
instead of relying on e820/"System RAM resources" - I have no clue.

>> I do agree that a "disable sysfs" option is interesting - even with
>> memory hotplug (we mostly need a way to configure it and a way to notify
>> kexec-tools about memory hot(un)plug events). I am currently (once
>> again) looking into improving auto-onlining support in the kernel.
>>
>> Having that said, I much rather want to see smaller improvements (that
>> can be fine-tuned individually - like allowing variable-sized memory
>> blocks) than doing a switch to "new shiny" and figuring out after a
>> while that we need "new shiny2".
> 
> There is only one certainty. Providing a long term interface with ever
> growing (ab)users is a hard target. And shinyN might be needed in the
> end. Who knows. My main point is that the existing interface is hitting
> a wall on usecases which _do_not_care_ about memory hotplug. And that is
> something we should be looking at.

Agreed. I can see 3 scenarios

a) no memory hotplug support, no sysfs.
b) memory hotplug support, no sysfs
c) memory hotplug support, sysfs

Starting with a) and c) is the easiest way to go.

> 
>> I consider removing "phys_device" as one of these tunables. The question
>> would be how to make such sysfs changes easy to configure
>> ("-phys_device", "+variable_sized_blocks" ...)
> 
> I am with you on that. There are more candidates in memory block
> directories which have dubious value. Deprecation process is a PITA and
> that's why I thought that it would make sense to focus on something that
> we can mis^Wdesign with exising and forming usecases in mind that would
> get rid of all the cruft that we know it doesn't work (removable would
> be another one.

Yeah, "phys_index" is also dubious. Simply providing a memory range
would have been much cleaner. Lesson learned :)

-- 
Thanks,

David / dhildenb