Re: [PATCH v3] mm: remove min_free_kbytes adjustment for THP

[Zi Yan]

>
> If you look at try_to_claim_block(), you need half of a block to be
> free or compatible with the requested migratetype in order to convert
> it. When memory is full, LRU pages are scattered all over, and
> compaction is not involved (order-0), this gets more difficult the
> bigger the block is. You can get into a situation where LRU reclaim
> will not clear sufficient room for conversion in any given pageblock
> anymore and you're stuck with the type distribution. A large share of
> buddy requests then go permanently through the slower fallback path.

I think we are digressing from the min_free_kbytes discussion, but it is
good to understand the limitations of our compaction and reclaim. Nimrod
is going to run some workloads to show us how his patch affect 4KB/2MB
THP systems. Based on the min_free_kbytes calculation, this patch has no
effect (on 4KB base page system) for a single node system with 272GB memory
and a two-node system with 484GB memory. So smaller machines would see
the impact of this patch. For example, a 16GB single node machine will
have 16MB min_free_kbytes instead of 66MB.

>
> Usama knows more about this, but we have seen this on GB300 hosts, and
> have JUST started to deploy kernels with smaller pageblocks (2M).
>
>> Basically it indicates at some point kernel allocates a lot of
>> unmovable pages that use many 512MB pageblocks and the life time of
>> these unmovable pages are so diverse, leading to all these
>> pageblocks remain unmovable and free pages spread across all these
>> pageblocks. I thought bigger pageblocks can keep unmovable pages
>> constrained within fewer pageblocks, leaving more contiguous free
>> memory.
>
> The idea is that the pageblock maintains contiguity for the largest
> size you routinely expect to allocate.
>
> The page allocator is very passive right now, and it doesn't work
> super reliably. But even in the current regime, smaller blocks have a
> better chance of containment.
>
> For example, when the ever-growing page cache runs out of movable
> block space, it spills into unmovable free space. When the next
> unmovable request finds no space, it runs LRU reclaim - which is more
> likely to free space in one of the many movable blocks. And so the
> next block is poisoned. Smaller blocks have a better chance of filling
> up natively, means less pressure to spill into incompatible ones.
>
> And the higher min_free_kbytes, the more likely there are still native
> options when the zones are down to the watermarks. E.g. better odds
> there is still unmovable free space, you just need to reclaim some
> movable/reclaimable space elsewhere to satisfy the watermarks.
>
> I've been working on making this more robust with the huge page
> allocator / defrag_mode stuff: instead of falling back and poisoning a
> block, invoke reclaim/compaction to produce a neutral block that can
> be converted entirely.
>
> It's the same idea as the higher min_free_kbytes and watermark
> boosting, but it is more targeted at the end result: readily available
> space in compatible or convertible blocks.
>
> But with that active regime, oversized pageblocks are even
> worse. You'd pay ongoing compaction work to produce a level of
> contiguity that you don't actually need.

OK, digress to the compaction and reclaim discussion. Isn't this 512MB
pageblock size issue also telling us 1GB THP allocation will not work at
all? It is a proxy of 1GB super-pageblock.

I need to spend some time on it to understand all the heuristics and see
if anything can be improved.

A quick chat with Claude results in some items for me to explore:

1. reduce pageblock conversion threshold from half of the pageblock.
That can reduce the number of slowpath entries.

2. make reclaim pageblock aware, so when an order-0 unmovable page
allocation incurs a reclaim, the reclaim will try to get free movable
pages from an unmovable pageblock first.

3. in addition to produce a neutral block, compaction/reclaim need to
repair unmovable blocks by getting rid of movable pages. Proactive
compaction should do that too.

>
>> > Seems to me the excessive min_free_kbytes is just a symptom of a
>> > deeper problem.
>> 
>> Yes, our anti-fragmentation mechanism does not work as we expected,
>> so that we need an excessive min_free_kbytes to get khugepaged working.
>> I wonder why reclaim cannot get the extra free memory instead of
>> reserving it via min_free_kbytes. Maybe we need a watermark boost
>> when some consecutive THP allocations are seen to achieve similar
>> effect of boosting min_free_kbytes?
>
> I'm just wondering what the easiest way forward is to fix the ARM 64k
> page problem.
>
> Yes, optimally, reclaim would work to satisfy compaction space by
> itself. We've seen it fail at that before, though.
>
> How critical set_recommended_min_free_kbytes() is today is a question
> that neither of us has a clear answer to. It's from 2011 and a lot has
> changed. However, knowing Andrea, I'm willing to bet he added this
> based on seeing a need in testing data. And I would actually expect it
> to work better now with proactive compaction, since that has a better
> chance of turning low-order chunks of that volume into pageblocks that
> can be converted instead of needing a poisoning steal.
>
> It's a change of long-standing behavior for everybody. It has a
> regression risk and requires careful evaluation and testing.

Not for 4KB systems with large memory, since khugepaged's
min_free_kbytes is capped at 66MB for one-node, 88MB for two-node, and
44MB + N*22MB for N-node. For a GB300 host you mentioned above, it has
494GB memory and with pageblock size set to 2MB, it should be a nop,
right?

>
> Meanwhile, adjusting the pageblock size on 64k page arm configs has a
> much smaller blast radius, appears to be the right move ANYWAY given
> what pageblocks are for, and makes the min_free_kbytes a non-issue.


-- 
Best Regards,
Yan, Zi