Re: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance

[Ryan Roberts]

On 07/07/2023 20:06, David Hildenbrand wrote:
>>>> I still feel that it would be better for the thp and large anon folio controls
>>>> to be independent though - what's the argument for tying them together?
>>>
>>> Thinking about desired 2 MiB flexible THP on aarch64 (64k kernel) vs, 2 MiB PMD
>>> THP on aarch64 (4k kernel), how are they any different? Just the way they are
>>> mapped ...
>>
>> The last patch in the series shows my current approach to that:
>>
>> int arch_wants_pte_order(struct vm_area_struct *vma)
>> {
>>     if (hugepage_vma_check(vma, vma->vm_flags, false, true, true))
>>         return CONFIG_ARM64_PTE_ORDER_THP; <<< always the contpte size
>>     else
>>         return CONFIG_ARM64_PTE_ORDER_NOTHP; <<< limited to 64K
>> }
>>
>> But Yu has raised concerns that this type of policy needs to be in the core mm.
>> So we could have the arch blindly return the preferred order from HW perspective
>> (which would be contpte size for arm64). Then for !hugepage_vma_check(), mm
>> could take the min of that value and some determined "acceptable" limit (which
>> in my mind is 64K ;-).
> 
> Yeah, it's really tricky. Because why should arm64 with 64k base pages *not*
> return 2MiB (which is one possible cont-pte size IIRC) ?
> 
> I share the idea that 64k might *currently* on *some platforms* be a reasonable
> choice. But that's where the "fun" begins.
> 
>>
>>>
>>> It's easy to say "64k vs. 2 MiB" is a difference and we want separate controls,
>>> but how is "2MiB vs. 2 MiB" different?
>>>
>>> Having that said, I think we have to make up our mind how much control we want
>>> to give user space. Again, the "2MiB vs. 2 MiB" case nicely shows that it's not
>>> trivial: memory waste is a real issue on some systems where we limit THP to
>>> madvise().
>>>
>>>
>>> Just throwing it out for discussing:
>>>
>>> What about keeping the "all / madvise / never" semantics (and MADV_NOHUGEPAGE
>>> ...) but having an additional config knob that specifies in which cases we
>>> *still* allow flexible THP even though the system was configured for "madvise".
>>>
>>> I can't come up with a good name for that, but something like
>>> "max_auto_size=64k" could be something reasonable to set. We could have an
>>> arch+hw specific default.
>>
>> Ahha, yes, that's essentially what I have above. I personally also like the idea
>> of the limit being an absolute value rather than an order. Although I know Yu
>> feels differently (see [1]).
> 
> Exposed to user space I think it should be a human-readable value. Inside the
> kernel, I don't particularly care.

My point was less about human-readable vs not. It was about expressing a value
that is relative to the base page size vs expressing a value that is independent
of base page size. If the concern is about limiting internal fragmentation, I
think its the absolute size that matters.

> 
> (Having databases/VMs on arch64 with 64k in mind) I think it might be
> interesting to have something like the following:
> 
> thp=madvise
> max_auto_size=64k/128k/256k
> 
> 
> So in MADV_HUGEPAGE VMAs (such as under QEMU), we'd happily take any flexible
> THP, especially ones < PMD THP (512 MiB) as well. 2 MiB or 4 MiB THP? sure, give
> them to my VM. You're barely going to find 512 MiB THP either way in practice ....
> 
> But for the remainder of my system, just do something reasonable and don't go
> crazy on the memory waste.

Yep, we're on the same page. I've got a v3 that's almost ready to go, based on
Yu's prevuous round of review. I'm going to encorporate this mechanism into it
then post hopefully later in the week. Now I just need to figure out a decent
name for the max_auto_size control...

> 
> 
> I'll try reading all the previous discussions next week.
>