Re: Correlation CMA size and FORCE_MAX_ZONEORDER

From: David Hildenbrand
Date: Mon Sep 19 2022 - 08:09:45 EST

Next message: Geert Uytterhoeven: "Re: [PATCH] clk: renesas: r9a07g043: Hide WDT2 clock and reset entries for RZ/Five"
Previous message: Yang Shen: "[RFC PATCH 4/6] crypto: benchmark - add help information"
In reply to: Michael Nazzareno Trimarchi: "Re: Correlation CMA size and FORCE_MAX_ZONEORDER"
Next in thread: Michael Nazzareno Trimarchi: "Re: Correlation CMA size and FORCE_MAX_ZONEORDER"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On 19.09.22 13:59, Michael Nazzareno Trimarchi wrote:

Hi David

On Mon, Sep 19, 2022 at 1:28 PM David Hildenbrand <david@xxxxxxxxxx> wrote:

On 19.09.22 13:17, Michael Nazzareno Trimarchi wrote:

Hi

On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@xxxxxxxxxx> wrote:

On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:

Hi

On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@xxxxxxxxxx> wrote:

On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:

Hi David

On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@xxxxxxxxxx> wrote:

On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:

Hi all

Hi,

Working on a small device with 128MB of memory and using imx_v6_v7
defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
are not respected. The calculation done does not allow the requested
size. I think that this should be somehow documented and described but
I did not
find the documentation. Does it work this way?

With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb

The underlying constraint is that CMA regions require a certain minimum
alignment+size. They cannot be arbitrarily in size.

CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
kernels to the size of a single pageblock.

In previous kernels, it used to be the size of the largest buddy
allocation granularity (derived from MAX_ORDER, derived from
FORCE_MAX_ZONEORDER).

On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
apply. On most archs, the minimum alignment+size should be 2 MiB
(x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.

So far the theory. Are you still running into this limitation on
upstream kernels?

I can run 6-rc2 on my board. I test again but according to it, if I
put 4M as CMA in cma=4M in boot
parameters, the result is 32Mb of CMA. Apart of that seems that
process lime tiny membench can not even start
to mblock memory

The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
4M, can you post /proc/meminfo and the early console output?

cat /proc/cmdline
cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
root=ubi0:root rw ubi.mtd=ro
otfs rootfstype=ubifs rootwait=1
# cat /proc/meminfo
MemTotal: 109560 kB
MemFree: 56084 kB
MemAvailable: 56820 kB
Buffers: 0 kB
Cached: 39680 kB
SwapCached: 0 kB
Active: 44 kB
Inactive: 644 kB
Active(anon): 44 kB
Inactive(anon): 644 kB
Active(file): 0 kB
Inactive(file): 0 kB
Unevictable: 39596 kB
Mlocked: 0 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 109560 kB
LowFree: 56084 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 628 kB
Mapped: 1480 kB
Shmem: 84 kB
KReclaimable: 4268 kB
Slab: 8456 kB
SReclaimable: 4268 kB
SUnreclaim: 4188 kB
KernelStack: 392 kB
PageTables: 88 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 54780 kB
Committed_AS: 1876 kB
VmallocTotal: 901120 kB
VmallocUsed: 2776 kB
VmallocChunk: 0 kB
Percpu: 72 kB
CmaTotal: 32768 kB
CmaFree: 32484 kB
# uname -a
Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
#

Then here https://pastebin.com/6MUB2VBM dmesg

CONFIG_ARM_MODULE_PLTS=y
CONFIG_FORCE_MAX_ZONEORDER=14
CONFIG_ALIGNMENT_TRAP=y
...
CONFIG_CMA
CONFIG_CMA_AREAS=7
...

CONFIG_CMA_SIZE_MBYTES=8
CONFIG_CMA_SIZE_SEL_MBYTES=y

Thanks!

I assume that in your setup, the pageblock size depends on MAX_ORDER
and, therefore, FORCE_MAX_ZONEORDER.

This should be the case especially if CONFIG_HUGETLB_PAGE is not defined
(include/linux/pageblock-flags.h).

In contrast to what I remember, the pageblock size does not seem to
depend on the THP size (weird) as well.

So, yes, that limitation is still in effect for some kernel configs.

One could make the pageblock size configurable (similar to
CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller
pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.

I imagine something reasonable might be to set the pageblock size to
2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.

I don't think making more configuration options makes things clear.

Yes, in an ideal case it should be automatic.

When we enable some configuration
we can force down the configuration. You need to explain clearly how
you envision it. FORCE_MAX_ZONEORDER
for me is the largest allocation that you can get from a zone (ex CMA
one). Any request allocation that is align to the

FORCE_MAX_ZONEORDER is just a way to increase/decrease the maximum
allocation size of the buddy in general.

CMA align and can fit inside a region should be allowed

What am I missing?

I think that the issue is that the CMA alignments nowadays depend on the
pageblock size. And the pageblock size depends on *some* configurations
on the maximum allocation size of the buddy.

Documenting the interaction between FORCE_MAX_ZONEORDER and CMA size
alignment is not trivial.

For example, with CONFIG_HUGETLB_PAGE there might not be such an
interaction. With CONFIG_HUGETLB_PAGE, there clearly is one.

Let me phrase it this way: is it good enough in you setup to get 32mb vs
8mb or do you want/need to reduce it without adjusting
FORCE_MAX_ZONEORDER ?

Wait we have:
- CMA kconfig alignment that in most config we have not considered
natural dma alignment but is put to 1Mb in a lot of embedded
- We have CMA_SIZE, CMA_SIZE_PERCENTAGE etc. Those seems that are not
effect if ZONEORDER is not reasonable and without
HUGETLB_PAGE
- etc

Changing MAX_ZONEORDER is ok and yes if you have an IOT device that
you know about your CMA allocation, it makes no sense to have
it 32MB for a 128Mb device. What I point out is that I need to figure
it out because in Kconfig there is no mention of it. Should it be
added there?

Something like

"Note that in some configurations, the CMA size must be aligned to the maximum allocation granularity of the buddy allocator, consequently depending on FORCE_MAX_ZONEORDER. The requested CMA size might get increased accordingly."

Maybe we'd want some kind of a warning in the kernel as well. If someone specifies "cma=2MB" but gets 32MB or more that might be a problem.

Does something like that make sense to you?

--
Thanks,

David / dhildenb

Next message: Geert Uytterhoeven: "Re: [PATCH] clk: renesas: r9a07g043: Hide WDT2 clock and reset entries for RZ/Five"
Previous message: Yang Shen: "[RFC PATCH 4/6] crypto: benchmark - add help information"
In reply to: Michael Nazzareno Trimarchi: "Re: Correlation CMA size and FORCE_MAX_ZONEORDER"
Next in thread: Michael Nazzareno Trimarchi: "Re: Correlation CMA size and FORCE_MAX_ZONEORDER"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]