Re: [LKP] [mm] ac5b2c1891: vm-scalability.throughput -61.3% regression

From: David Rientjes
Date: Tue Dec 04 2018 - 19:07:33 EST


On Tue, 4 Dec 2018, Michal Hocko wrote:

> The thing I am really up to here is that reintroduction of
> __GFP_THISNODE, which you are pushing for, will conflate madvise mode
> resp. defrag=always with a numa placement policy because the allocation
> doesn't fallback to a remote node.
>

It isn't specific to MADV_HUGEPAGE, it is the policy for all transparent
hugepage allocations, including defrag=always. We agree that
MADV_HUGEPAGE is not exactly defined: does it mean try harder to allocate
a hugepage locally, try compaction synchronous to the fault, allow remote
fallback? It's undefined.

The original intent was to be used when thp is disabled system wide
(enabled set to "madvise") because its possible the rss of the process
increases if backed by thp. That occurs either if faulting on a hugepage
aligned area or based on max_ptes_none. So we have at least three
possible policies that have evolved over time: preventing increased rss,
direct compaction, remote fallback. Certainly not something that fits
under a single madvise mode.

> And that is a fundamental problem and the antipattern I am talking
> about. Look at it this way. All normal allocations are utilizing all the
> available memory even though they might hit a remote latency penalty. If
> you do care about NUMA placement you have an API to enforce a specific
> placement. What is so different about THP to behave differently. Do
> we really want to later invent an API to actually allow to utilize all
> the memory? There are certainly usecases (that triggered the discussion
> previously) that do not mind the remote latency because all other
> benefits simply outweight it?
>

What is different about THP is that on every platform I have measured,
NUMA matters more than hugepages. Obviously if on Broadwell, Haswell, and
Rome, remote hugepages were a performance win over local pages, this
discussion would not be happening. Faulting local pages rather than
local hugepages, if possible, is easy and doesn't require reclaim.
Faulting remote pages rather than reclaiming local pages is easy in your
scenario, it's non-disruptive.

So to answer "what is so different about THP?", it's the performance data.
The NUMA locality matters more than whether the pages are huge or not. We
also have the added benefit of khugepaged being able to collapse pages
locally if fragmentation improves rather than being stuck accessing a
remote hugepage forever.

> That being said what should users who want to use all the memory do to
> use as many THPs as possible?

If those users want to accept the performance degradation of allocating
remote hugepages instead of local pages, that should likely be an
extension, either madvise or prctl. That's not necessarily the usecase
Andrea would have, I don't believe: he'd still prefer to compact memory
locally and avoid the swap storm than allocate remotely. If impossible to
reclaim locally for regular pages, remote hugepages may be more beneficial
than remote pages.