Re: [LSF/MM ATTEND ] memory reclaim with NUMA rebalancing

From: Larry Woodman
Date: Fri Feb 22 2019 - 09:12:18 EST


On 02/06/2019 02:03 PM, Christopher Lameter wrote:
> On Thu, 31 Jan 2019, Aneesh Kumar K.V wrote:
>
>> I would be interested in this topic too. I would like to
>> understand the API and how it can help exploit the different type of
>> devices we have on OpenCAPI.
Same here, we/RedHat have quite a bit of experience running on several
large system
(32TB/128nodes/1024CPUs). Some of these systems have NVRAM and can operated
in memory mode as well as storage mode.

Larry

> So am I. We may want to rethink the whole NUMA API and the way we handle
> different types of memory with their divergent performance
> characteristics.
>
> We need some way to allow a better selection of memory from the kernel
> without creating too much complexity. We have new characteristics to
> cover:
>
> 1. Persistence (NVRAM) or generally a storage device that allows access to
> the medium via a RAM like interface.
>
> 2. Coprocessor memory that can be shuffled back and forth to a device
> (HMM).
>
> 3. On Device memory (important since PCIe limitations are currently a
> problem and Intel is stuck on PCIe3 and devices start to bypass the
> processor to gain performance)
>
> 4. High Density RAM (GDDR f.e.) with different caching behavior
> and/or different cacheline sizes.
>
> 5. Modifying access characteristics by reserving slice of a cache (f.e.
> L3) for a specific memory region.
>
> 6. SRAM support (high speed memory on the processor itself or by using
> the processor cache to persist a cacheline)
>
> And then the old NUMA stuff where only the latency to memory varies. But
> that was a particular solution targeted at scaling SMP system through
> interconnects. This was a mostly symmetric approach. The use of
> accellerators etc etc and the above characteristics lead to more complex
> assymmetric memory approaches that may be difficult to manage and use from
> kernel space.
>