Re: THP backed thread stacks

[Zach O'Keefe]

On Wed, Mar 8, 2023 at 11:02 AM Mike Kravetz <mike.kravetz@xxxxxxxxxx> wrote:
>
> On 03/06/23 16:40, Mike Kravetz wrote:
> > On 03/06/23 19:15, Peter Xu wrote:
> > > On Mon, Mar 06, 2023 at 03:57:30PM -0800, Mike Kravetz wrote:
> > > >
> > > > Just wondering if there is anything better or more selective that can be
> > > > done?  Does it make sense to have THP backed stacks by default?  If not,
> > > > who would be best at disabling?  A couple thoughts:
> > > > - The kernel could disable huge pages on stacks.  libpthread/glibc pass
> > > >   the unused flag MAP_STACK.  We could key off this and disable huge pages.
> > > >   However, I'm sure there is somebody somewhere today that is getting better
> > > >   performance because they have huge pages backing their stacks.
> > > > - We could push this to glibc/libpthreads and have them use
> > > >   MADV_NOHUGEPAGE on thread stacks.  However, this also has the potential
> > > >   of regressing performance if somebody somewhere is getting better
> > > >   performance due to huge pages.
> > >
> > > Yes it seems it's always not safe to change a default behavior to me.
> > >
> > > For stack I really can't tell why it must be different here.  I assume the
> > > problem is the wasted space and it exaggerates easily with N-threads.  But
> > > IIUC it'll be the same as thp to normal memories iiuc, e.g., there can be a
> > > per-thread mmap() of 2MB even if only 4K is used each, then if such mmap()
> > > is populated by THP for each thread there'll also be a huge waste.
>
> I may be alone in my thinking here, but it seems that stacks by their nature
> are not generally good candidates for huge pages.  I am just thinking about
> the 'normal' use case where stacks contain local function data and arguments.
> Am I missing something, or are huge pages really a benefit here?
>
> Of course, I can imagine some thread with a large amount of frequently
> accessed data allocated on it's stack which could benefit from huge
> pages.  But, this seems to be an exception rather than the rule.
>
> I understand the argument that THP always means always and everywhere.
> It just seems that thread stacks may be 'special enough' to consider
> disabling by default

Just my drive-by 2c, but would agree with you here (at least wrt
hugepages not being good candidates, in general). A user mmap()'ing
memory has a lot more (direct) control over how they fault / utilize
the memory: you know when you're running out of space and can map more
space as needed. For these stacks, you're setting the stack size to
2MB just as a precaution so you can avoid overflow -- AFAIU there is
no intention of using the whole mapping (and looking at some data,
it's very likely you won't come close).

That said, why bother setting stack attribute to 2MiB in size if there
isn't some intention of possibly being THP-backed? Moreover, how did
it happen that the mappings were always hugepage-aligned here?

> --
> Mike Kravetz
>