Re: CXL Boot to Bash - Section 3: Memory (block) Hotplug

[Gregory Price]

On Tue, Mar 11, 2025 at 12:08:03PM -0400, Zi Yan wrote:
> On 11 Mar 2025, at 11:58, Gregory Price wrote:
> 
> > On Tue, Mar 11, 2025 at 10:53:41AM -0400, Zi Yan wrote:
> >> On 20 Feb 2025, at 14:26, David Hildenbrand wrote:
> >>
> >>> Yes, the only way to get some 1 GiB pages is by using larger memory blocks (e.g., 2 GiB on x86-64), which comes with a different set of issues (esp. hotplug granularity).
> >>
> >> An alternative I can think of is to mark a hot-plugged memory block dedicated
> >> to memmap and use it for new memory block’s memmap provision. In this way,
> >> a 256MB memory block can be used for 256MB*(256MB/4MB)=16GB hot plugged memory.
> >> Yes, it will waste memory before 256MB+16GB is online, but that might be
> >> easier to handle than variable sized memory block, I suppose?
> >>
> >>>
> >
> > The devil is in the details here.  We'd need a way for the driver to
> > tell hotplug "use this for memmap for some yet-to-be-mapped region" -
> > rather than having that allocate naturally.  Either this, or a special
> > ZONE specifically for memmap allocations.
> 
> Or a new option for memmap_on_memory like “use_whole_block”, then hotplug
> code checks altmap is NULL or not when a memory block is plugged.
> If altmap is NULL, the hot plugged memory block is used as memmap,
> otherwise, the memory block is plugged as normal memory. The code also
> needs to maintain which part of the altmap is used to tell whether
> the memmap’d memory block can be offline or not.
>

Also just to be clear, this is only an issue if you intend to use CXL
memory for something like 1GB Gigantic pages - which do not support
ZONE_MOVABLE anyway.  So for this to matter your system must:

1) Require smaller than 1GB alignment for memblocks, and
2) ZONE_NORMAL CXL memory.

The whole thing is mitigated by telling your platform vendor to align
memory base to 2GB and having DCD's allocate in 2GB aligned extents.

Both of these are reasonable requirements for hardware, requiring a
major kernel reconfiguration seems less reasonable.

We should continue talking, but I've reached the conclusion that simply
telling platform vendors to fix their alignment is a better overall
solution.

~Gregory