Re: [PATCH 00/21] mm: introduce Designated Movable Blocks

From: Doug Berger
Date: Fri Sep 23 2022 - 18:10:54 EST


On 9/23/2022 4:19 AM, Mike Rapoport wrote:
Hi Doug,

I only had time to skim through the patches and before diving in I'd like
to clarify a few things.
Thanks for taking the time. Any input is appreciated.


On Mon, Sep 19, 2022 at 06:03:55PM -0700, Doug Berger wrote:
On 9/19/2022 2:00 AM, David Hildenbrand wrote:

How is this memory currently presented to the system?

The 7278 device has four ARMv8 CPU cores in an SMP cluster and two memory
controllers (MEMCs). Each MEMC is capable of controlling up to 8GB of DRAM.
An example 7278 system might have 1GB on each controller, so an arm64 kernel
might see 1GB on MEMC0 at 0x40000000-0x7FFFFFFF and 1GB on MEMC1 at
0x300000000-0x33FFFFFFF.

The base capability described in commits 7-15 of this V1 patch set is to
allow a 'movablecore' block to be created at a particular base address
rather than solely at the end of addressable memory.

I think this capability is only useful when there is non-uniform access to
different memory ranges. Otherwise it wouldn't matter where the movable
pages reside.
I think that is a fair assessment of the described capability. However, the non-uniform access is a result of the current Linux architecture rather than the hardware architecture.

The system you describe looks quite NUMA to me, with two
memory controllers, each for accessing a partial range of the available
memory.
NUMA was created to deal with non-uniformity in the hardware architecture where a CPU and/or other hardware device can make more efficient use of some nodes than other nodes. NUMA attempts to allocate from "closer" nodes to improve the operational efficiency of the system.

If we consider how an arm64 architecture Linux kernel will apply zones to the above example system we find that Linux will place MEMC0 in ZONE_DMA and MEMC1 in ZONE_NORMAL. This allows both kernel and user space to compete for bandwidth on MEMC1, but largely excludes user space from MEMC0. It is possible for user space to get memory from ZONE_DMA through fallback when ZONE_NORMAL has been consumed, but there is a pretty clear bias against user space use of MEMC0. This non-uniformity doesn't come from the bus architecture since each CPU has equal costs to access MEMC0 and MEMC1. They compete for bandwidth, but there is no hardware bias for one node over another. Creating ZONE_MOVABLE memory on MEMC0 can help correct for the Linux bias.

expressed the desire to locate ZONE_MOVABLE memory on each
memory controller to allow user space intensive processing to
make better use of the additional memory bandwidth.

Can you share some more how exactly ZONE_MOVABLE would help here to make
better use of the memory bandwidth?

ZONE_MOVABLE memory is effectively unusable by the kernel. It can be used by
user space applications through both the page allocator and the Hugetlbfs.
If a large 'movablecore' allocation is defined and it can only be located at
the end of addressable memory then it will always be located on MEMC1 of a
7278 system. This will create a tendency for user space accesses to consume
more bandwidth on the MEMC1 memory controller and kernel space accesses to
consume more bandwidth on MEMC0. A more even distribution of ZONE_MOVABLE
memory between the available memory controllers in theory makes more memory
bandwidth available to user space intensive loads.

The theory makes perfect sense, but is there any practical evidence of
improvement?
Some benchmark results that illustrate the difference would be nice.
I agree that benchmark results would be nice. Unfortunately, I am not part of the constituency that uses these Linux features so I have no representative user space work loads to measure. I can only say that I was asked to implement this capability, this is the approach I took, and customers of Broadcom are making use of it. I am submitting it upstream with the hope that: its/my sanity can be better reviewed, it will not get broken by future changes in the kernel, and it will be useful to others.

This "narrow" capability may have limited value to others, but it should not create issues for those that do not actively wish to use it. I would hope that makes it easier to review and get accepted.

However, I believe "other opportunities" exist that may have broader appeal so I have suggested some along with the "narrow" capability to hopefully give others motivation to consider accepting the narrow capability and to help shape how these "other capabilities" should be implemented.

One "other opportunity" that I have realized may be more interesting than I originally anticipated comes from the recognition that the Devicetree Specification includes support for Reserved Memory regions that can contain the 'reusable' property to allow the OS to make use of the memory. Currently, Linux only takes advantage of that capability for reserved memory nodes that are compatible with 'shared-dma-pool' where CMA is used to allow the memory to be used by the OS and by device drivers. CMA is a great concept, but we have observed shortcomings that become more apparent as the size of the CMA region grows. Specifically, the Linux memory management works very hard to keep half of the CMA memory free. A number of submissions have been made over the years to alter the CMA implementation to allow more aggressive use of the memory by the OS, but there are users that desire the current behavior so the submissions have been rejected.

No other types of reserved memory nodes can take advantage of sharing the memory with the Linux operating system because there is insufficient specification of how device drivers can reclaim the reserved memory when it is needed. The introduction of Designated Movable Block support provides a mechanism that would allow this capability to be realized. Because DMBs are in ZONE_MOVABLE their pages are reclaimable, and because they can be located anywhere they can satisfy DMA constraints of owning devices. In the simplest case, device drivers can use the dmb_intersects() function to determine whether their reserved memory range is within a DMB and can use the alloc_contig_range() function to reclaim the pages. This simple API could certainly be improved upon (e.g. the CMA allocator seems like an obvious choice), but it doesn't need to be defined by me so I would be happy to hear other people's ideas.


BACKGROUND:
NUMA architectures support distributing movablecore memory
across each node, but it is undesirable to introduce the
overhead and complexities of NUMA on systems that don't have a
Non-Uniform Memory Architecture.

How exactly would that look like? I think I am missing something :)

The notion would be to consider each memory controller as a separate node,
but as stated it is not desirable.

Why?
In my opinion this is an inappropriate application of NUMA because the hardware does not impose any access non-uniformity to justify the complexity and overhead associated with NUMA. It would only be shoe-horned into the implementation to add some logical notion of memory nodes being associated with memory controllers. I would expect such an approach to receive a lot of push back from the Android Common Kernel users which may not be relevant to everyone, but is to many.

Thanks for your consideration,
-Doug