Re: [RFC 0/2] mm: introduce THP deferred setting

[Nico Pache]

Hi Zi Yan,
On Mon, Jul 29, 2024 at 7:26 PM Zi Yan <ziy@xxxxxxxxxx> wrote:
>
> +Kirill
>
> On 29 Jul 2024, at 18:27, Nico Pache wrote:
>
> > We've seen cases were customers switching from RHEL7 to RHEL8 see a
> > significant increase in the memory footprint for the same workloads.
> >
> > Through our investigations we found that a large contributing factor to
> > the increase in RSS was an increase in THP usage.
>
> Any knob is changed from RHEL7 to RHEL8 to cause more THP usage?
IIRC, most of the systems tuning is the same. We attributed the
increase in THP usage to a combination of improvements in the kernel,
and improvements in the libraries (better alignments). That allowed
THP allocations to succeed at a higher rate. I can go back and confirm
this tomorrow though.
>
> >
> > For workloads like MySQL, or when using allocators like jemalloc, it is
> > often recommended to set /transparent_hugepages/enabled=never. This is
> > in part due to performance degradations and increased memory waste.
> >
> > This series introduces enabled=defer, this setting acts as a middle
> > ground between always and madvise. If the mapping is MADV_HUGEPAGE, the
> > page fault handler will act normally, making a hugepage if possible. If
> > the allocation is not MADV_HUGEPAGE, then the page fault handler will
> > default to the base size allocation. The caveat is that khugepaged can
> > still operate on pages thats not MADV_HUGEPAGE.
>
> Why? If user does not explicitly want huge page, why bother providing huge
> pages? Wouldn't it increase memory footprint?

So we have "always", which will always try to allocate a THP when it
can. This setting gives good performance in a lot of conditions, but
tends to waste memory. Additionally applications DON'T need to be
modified to take advantage of THPs.

We have "madvise" which will only satisfy allocations that are
MADV_HUGEPAGE, this gives you granular control, and a lot of times
these madvises come from libraries. Unlike "always" you DO need to
modify your application if you want to use THPs.

Then we have "never", which of course, never allocates THPs.

Ok. back to your question, like "madvise", "defer" gives you the
benefits of THPs when you specifically know you want them
(madv_hugepage), but also benefits applications that dont specifically
ask for them (or cant be modified to ask for them), like "always"
does. The applications that dont ask for THPs must wait for khugepaged
to get them (avoid insertions at PF time)-- this curbs a lot of memory
waste, and gives an increased tunability over "always". Another added
benefit is that khugepaged will most likely not operate on short lived
allocations, meaning that only longstanding memory will be collapsed
to THPs.

The memory waste can be tuned with max_ptes_none... lets say you want
~90% of your PMD to be full before collapsing into a huge page. simply
set max_ptes_none=64. or no waste, set max_ptes_none=0, requiring the
512 pages to be present before being collapsed.

>
> >
> > This allows for two things... one, applications specifically designed to
> > use hugepages will get them, and two, applications that don't use
> > hugepages can still benefit from them without aggressively inserting
> > THPs at every possible chance. This curbs the memory waste, and defers
> > the use of hugepages to khugepaged. Khugepaged can then scan the memory
> > for eligible collapsing.
>
> khugepaged would replace application memory with huge pages without specific
> goal. Why not use a user space agent with process_madvise() to collapse
> huge pages? Admin might have more knobs to tweak than khugepaged.

The benefits of "always" are that no userspace agent is needed, and
applications dont have to be modified to use madvise(MADV_HUGEPAGE) to
benefit from THPs. This setting hopes to gain some of the same
benefits without the significant waste of memory and an increased
tunability.

future changes I have in the works are to make khugepaged more
"smart". Moving it away from the round robin fashion it currently
operates in, to instead make smart and informed decisions of what
memory to collapse (and potentially split).

Hopefully that helped explain the motivation for this new setting!

Cheer!
-- Nico
>
> >
> > Admins may want to lower max_ptes_none, if not, khugepaged may
> > aggressively collapse single allocations into hugepages.
> >
> > RFC note
> > ==========
> > Im not sure if im missing anything related to the mTHP
> > changes. I think now that we have hugepage_pmd_enabled in
> > commit 00f58104202c ("mm: fix khugepaged activation policy") everything
> > should work as expected.
> >
> > Nico Pache (2):
> >   mm: defer THP insertion to khugepaged
> >   mm: document transparent_hugepage=defer usage
> >
> >  Documentation/admin-guide/mm/transhuge.rst | 18 ++++++++++---
> >  include/linux/huge_mm.h                    | 15 +++++++++--
> >  mm/huge_memory.c                           | 31 +++++++++++++++++++---
> >  3 files changed, 55 insertions(+), 9 deletions(-)
> >
> > Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
> > Cc: David Hildenbrand <david@xxxxxxxxxx>
> > Cc: Matthew Wilcox <willy@xxxxxxxxxxxxx>
> > Cc: Barry Song <baohua@xxxxxxxxxx>
> > Cc: Ryan Roberts <ryan.roberts@xxxxxxx>
> > Cc: Baolin Wang <baolin.wang@xxxxxxxxxxxxxxxxx>
> > Cc: Lance Yang <ioworker0@xxxxxxxxx>
> > Cc: Peter Xu <peterx@xxxxxxxxxx>
> > Cc: Zi Yan <ziy@xxxxxxxxxx>
> > Cc: Rafael Aquini <aquini@xxxxxxxxxx>
> > Cc: Andrea Arcangeli <aarcange@xxxxxxxxxx>
> > Cc: Jonathan Corbet <corbet@xxxxxxx>
> > --
> > 2.45.2
>
> --
> Best Regards,
> Yan, Zi