Re: Correlation CMA size and FORCE_MAX_ZONEORDER

From: Michael Nazzareno Trimarchi
Date: Mon Sep 19 2022 - 07:18:04 EST


Hi

On Mon, Sep 19, 2022 at 1:03 PM David Hildenbrand <david@xxxxxxxxxx> wrote:
>
> On 19.09.22 11:57, Michael Nazzareno Trimarchi wrote:
> > Hi
> >
> > On Mon, Sep 19, 2022 at 11:31 AM David Hildenbrand <david@xxxxxxxxxx> wrote:
> >>
> >> On 19.09.22 11:17, Michael Nazzareno Trimarchi wrote:
> >>> Hi David
> >>>
> >>> On Mon, Sep 19, 2022 at 10:38 AM David Hildenbrand <david@xxxxxxxxxx> wrote:
> >>>>
> >>>> On 15.09.22 23:36, Michael Nazzareno Trimarchi wrote:
> >>>>> Hi all
> >>>>
> >>>> Hi,
> >>>>
> >>>>>
> >>>>> Working on a small device with 128MB of memory and using imx_v6_v7
> >>>>> defconfig I found that CMA_SIZE_MBYTES, CMA_SIZE_PERCENTAGE
> >>>>> are not respected. The calculation done does not allow the requested
> >>>>> size. I think that this should be somehow documented and described but
> >>>>> I did not
> >>>>> find the documentation. Does it work this way?
> >>>>>
> >>>>> With CMA_SIZE of 8MB I need to have FORCE_MAX_ZONEORDER=12 if I have
> >>>>> the default FORCE_MAX_ZONEORDER=14 the min size is 32Mb
> >>>>
> >>>> The underlying constraint is that CMA regions require a certain minimum
> >>>> alignment+size. They cannot be arbitrarily in size.
> >>>>
> >>>> CMA_MIN_ALIGNMENT_BYTES expresses that, and corresponds in upstream
> >>>> kernels to the size of a single pageblock.
> >>>>
> >>>> In previous kernels, it used to be the size of the largest buddy
> >>>> allocation granularity (derived from MAX_ORDER, derived from
> >>>> FORCE_MAX_ZONEORDER).
> >>>>
> >>>> On upstream kernels, the FORCE_MAX_ZONEORDER constraint should no longer
> >>>> apply. On most archs, the minimum alignment+size should be 2 MiB
> >>>> (x86-64, aarch64 with 4k base pages) -- the size of a single pageblock.
> >>>>
> >>>> So far the theory. Are you still running into this limitation on
> >>>> upstream kernels?
> >>>>
> >>>
> >>> I can run 6-rc2 on my board. I test again but according to it, if I
> >>> put 4M as CMA in cma=4M in boot
> >>> parameters, the result is 32Mb of CMA. Apart of that seems that
> >>> process lime tiny membench can not even start
> >>> to mblock memory
> >>>
> >>
> >> The CMA alignemnt change went into v5.19. If "cma=4M" still gives you >
> >> 4M, can you post /proc/meminfo and the early console output?
> >>
> >
> > cat /proc/cmdline
> > cma=4M mtdparts=gpmi-nand:4m(nandboot),1m(env),24m(kernel),1m(nanddtb),-(rootfs)
> > root=ubi0:root rw ubi.mtd=ro
> > otfs rootfstype=ubifs rootwait=1
> > # cat /proc/meminfo
> > MemTotal: 109560 kB
> > MemFree: 56084 kB
> > MemAvailable: 56820 kB
> > Buffers: 0 kB
> > Cached: 39680 kB
> > SwapCached: 0 kB
> > Active: 44 kB
> > Inactive: 644 kB
> > Active(anon): 44 kB
> > Inactive(anon): 644 kB
> > Active(file): 0 kB
> > Inactive(file): 0 kB
> > Unevictable: 39596 kB
> > Mlocked: 0 kB
> > HighTotal: 0 kB
> > HighFree: 0 kB
> > LowTotal: 109560 kB
> > LowFree: 56084 kB
> > SwapTotal: 0 kB
> > SwapFree: 0 kB
> > Dirty: 0 kB
> > Writeback: 0 kB
> > AnonPages: 628 kB
> > Mapped: 1480 kB
> > Shmem: 84 kB
> > KReclaimable: 4268 kB
> > Slab: 8456 kB
> > SReclaimable: 4268 kB
> > SUnreclaim: 4188 kB
> > KernelStack: 392 kB
> > PageTables: 88 kB
> > NFS_Unstable: 0 kB
> > Bounce: 0 kB
> > WritebackTmp: 0 kB
> > CommitLimit: 54780 kB
> > Committed_AS: 1876 kB
> > VmallocTotal: 901120 kB
> > VmallocUsed: 2776 kB
> > VmallocChunk: 0 kB
> > Percpu: 72 kB
> > CmaTotal: 32768 kB
> > CmaFree: 32484 kB
> > # uname -a
> > Linux buildroot 6.0.0-rc5 #20 SMP Mon Sep 19 11:51:26 CEST 2022 armv7l GNU/Linux
> > #
> >
> > Then here https://pastebin.com/6MUB2VBM dmesg
> >
> > CONFIG_ARM_MODULE_PLTS=y
> > CONFIG_FORCE_MAX_ZONEORDER=14
> > CONFIG_ALIGNMENT_TRAP=y
> > ...
> > CONFIG_CMA
> > CONFIG_CMA_AREAS=7
> > ...
> >
> > CONFIG_CMA_SIZE_MBYTES=8
> > CONFIG_CMA_SIZE_SEL_MBYTES=y
> >
>
> Thanks!
>
> I assume that in your setup, the pageblock size depends on MAX_ORDER
> and, therefore, FORCE_MAX_ZONEORDER.
>
> This should be the case especially if CONFIG_HUGETLB_PAGE is not defined
> (include/linux/pageblock-flags.h).
>
> In contrast to what I remember, the pageblock size does not seem to
> depend on the THP size (weird) as well.
>
>
> So, yes, that limitation is still in effect for some kernel configs.
>
> One could make the pageblock size configurable (similar to
> CONFIG_HUGETLB_PAGE_SIZE_VARIABLE) or simply default to a smaller
> pageblock size as default with CONFIG_CMA and !CONFIG_HUGETLB_PAGE.
>
> I imagine something reasonable might be to set the pageblock size to
> 2MiB without CONFIG_HUGETLB_PAGE but with CONFIG_CMA.
>

I don't think making more configuration options makes things clear.
When we enable some configuration
we can force down the configuration. You need to explain clearly how
you envision it. FORCE_MAX_ZONEORDER
for me is the largest allocation that you can get from a zone (ex CMA
one). Any request allocation that is align to the
CMA align and can fit inside a region should be allowed

What am I missing?

Michael

> --
> Thanks,
>
> David / dhildenb
>
--
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@xxxxxxxxxxxxxxxxxxxx
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@xxxxxxxxxxxxxxxxxxxx
www.amarulasolutions.com