Re: CXL Boot to Bash - Section 3: Memory (block) Hotplug

[Gregory Price]

On Tue, Feb 18, 2025 at 08:25:59PM +0100, David Hildenbrand wrote:
> On 18.02.25 19:04, Gregory Price wrote:
> 
> Hm?
> 
> If you enable memmap_on_memory, we will place the memmap on that carved-out
> region, independent of ZONE_NORMAL/ZONE_MOVABLE etc. It's the "altmap".
>
> Reason that we can place the memmap on a ZONE_MOVABLE is because, although
> it is "unmovable", we told memory offlining code that it doesn't have to
> care about offlining that memmap carveout, there is no migration to be done.
> Just offline the block (memmap gets stale) and remove that block (memmap
> gets removed).
> 
> If there is a reason where we carve out the memmap and *not* use it, that
> case must be fixed.
> 

Hm, I managed to trace down the wrong path on this particular code.

I will go back and redo my tests to sanity check, but here's what I
would expect to see:

1) if memmap_on_memory is off, and hotplug capacity (node1) is
   zone_movable - then zone_normal (node0) should have N pages
   accounted in nr_memmap_pages

   1a) when dropping these memory blocks, I should see node0 memory
       use drop by 4GB - since this is just GFP_KERNEL pages.

2) if memmap_on_memory is on, and hotplug capacity (node1) is
   zone_movable - then each memory block (256MB) should appear
   as 252MB (-4MB of 64-byte page structs).  For 256GB (my system)
   I should see a total of 252GB of onlined memory (-4GB of page struct)

   2a) when dropping these memory blocks, I should see node0 memory use
       stay the same - since it was vmemmap usage.

I will double check that this isn't working as expected, and i'll double
check for a build option as well.

stupid question - it sorta seems like you'd want this as the default
setting for driver-managed hotplug memory blocks, but I suppose for
very small blocks there's problems (as described in the docs).

:thinking: - is it silly to suggest maybe a per-driver memmap_on_memory
setting rather than just a global setting?  For CXL capacity, this seems
like a no-brainer since blocks can't be smaller than 256MB (per spec).

~Gregory