RE: [PATCH v8 0/2] Dynamic Allocation of the reserved_mem array

[Aisheng Dong]

> From: Oreoluwa Babatunde <quic_obabatun@xxxxxxxxxxx>
> Sent: 2024年8月31日 0:29
> Subject: [PATCH v8 0/2] Dynamic Allocation of the reserved_mem array
> 
> The reserved_mem array is used to store data for the different reserved
> memory regions defined in the DT of a device.  The array stores information
> such as region name, node reference, start-address, and size of the different
> reserved memory regions.
> 
> The array is currently statically allocated with a size of
> MAX_RESERVED_REGIONS(64). This means that any system that specifies a
> number of reserved memory regions greater than
> MAX_RESERVED_REGIONS(64) will not have enough space to store the
> information for all the regions.
> 
> This can be fixed by making the reserved_mem array a dynamically sized array
> which is allocated using memblock_alloc() based on the exact number of
> reserved memory regions defined in the DT.
> 
> On architectures such as arm64, memblock allocated memory is not writable
> until after the page tables have been setup.
> This is an issue because the current implementation initializes the reserved
> memory regions and stores their information in the array before the page
> tables are setup. Hence, dynamically allocating the reserved_mem array and
> attempting to write information to it at this point will fail.
> 
> Therefore, the allocation of the reserved_mem array will need to be done after
> the page tables have been setup, which means that the reserved memory
> regions will also need to wait until after the page tables have been setup to be
> stored in the array.
> 
> When processing the reserved memory regions defined in the DT, these regions
> are marked as reserved by calling memblock_reserve(base, size).
> Where:  base = base address of the reserved region.
> 	size = the size of the reserved memory region.
> 
> Depending on if that region is defined using the "no-map" property,
> memblock_mark_nomap(base, size) is also called.
> 
> The "no-map" property is used to indicate to the operating system that a
> mapping of the specified region must NOT be created. This also means that no
> access (including speculative accesses) is allowed on this region of memory
> except when it is coming from the device driver that this region of memory is
> being reserved for.[1]
> 
> Therefore, it is important to call memblock_reserve() and
> memblock_mark_nomap() on all the reserved memory regions before the
> system sets up the page tables so that the system does not unknowingly
> include any of the no-map reserved memory regions in the memory map.
> 
> There are two ways to define how/where a reserved memory region is placed
> in memory:
> i) Statically-placed reserved memory regions i.e. regions defined with a set
> start address and size using the
>      "reg" property in the DT.
> ii) Dynamically-placed reserved memory regions.
> i.e. regions defined by specifying a range of addresses where they can
>      be placed in memory using the "alloc_ranges" and "size" properties
>      in the DT.
> 
> The dynamically-placed reserved memory regions get assigned a start address
> only at runtime. And this needs to  be done before the page tables are setup
> so that memblock_reserve() and memblock_mark_nomap() can be called on
> the allocated region as explained above.
> Since the dynamically allocated reserved_mem array can only be available
> after the page tables have been setup, the information for the
> dynamically-placed reserved memory regions needs to be stored somewhere
> temporarily until the reserved_mem array is available.
> 
> Therefore, this series makes use of a temporary static array to store the
> information of the dynamically-placed reserved memory regions until the
> reserved_mem array is allocated.
> Once the reserved_mem array is available, the information is copied over from
> the temporary array into the reserved_mem array, and the memory for the
> temporary array is freed back to the system.
> 
> The information for the statically-placed reserved memory regions does not
> need to be stored in a temporary array because their starting address is
> already stored in the devicetree.
> Once the reserved_mem array is allocated, the information for the
> statically-placed reserved memory regions is added to the array.
> 

I tested with MX8ULP that remoteproc became unwork after applying this patchset.
The same issue exist in linux-next with tag next-20240819.

Root cause is that this patchset breaks the API of_reserved_mem_device_init_by_idx()
used by coherent dma (kernel/dma/contiguous.c) due to rmem->ops was not
properly saved in fdt_init_reserved_mem_node() after calling reserved memory
setup function. e.g. rmem_dma_setup.

Regards
Aisheng

> Note:
> Because of the use of a temporary array to store the information of the
> dynamically-placed reserved memory regions, there still exists a limitation of
> 64 for this particular kind of reserved memory regions.
> From my observation, these regions are typically small in number and hence I
> expect this to not be an issue for now.
> 
> Patch Versions:
> v8:
> - Check the value of initial_boot_params in
>   fdt_scan_reserved_mem_reg_nodes() to avoid breakage on architectures
>   where this is not being used as was found to be the case for x86 in
>   the issues reported below: