Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 9/9/26 04:33, Zi Yan wrote:
> On Mon Sep 7, 2026 at 3:30 AM EDT, Vlastimil Babka (SUSE) wrote:
>> On 9/4/26 17:08, Zi Yan wrote:
>>> On Fri Sep 4, 2026 at 10:11 AM EDT, Vlastimil Babka (SUSE) wrote:
>>>> On 9/4/26 13:56, Salvatore Dipietro wrote:
>>>>> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>>>> introduced high-order folio allocations in the iomap buffered write
>>>>> path.  When memory is fragmented, each failed costly-order allocation
>>>>> enters __alloc_pages_slowpath() which runs direct compaction and
>>>>> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
>>>>> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
>>>>> 
>>>>> The root issue is that direct compaction is too expensive for hot
>>>>> allocation paths that have fallbacks to smaller allocations.
>>>>> __filemap_get_folio_mpol() already marks higher-order allocations with
>>>>> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
>>>>> failure.  However, the page allocator still attempts full direct
>>>>> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
>>>>> aggressive when the caller will simply retry at a lower order.
>>>>> 
>>>>> For costly-order allocations with __GFP_NORETRY, clear
>>>>> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
>>>>> can_direct_reclaim, can_compact and the nofail checks are evaluated.
>>>>> This makes the entire slowpath treat the request as non-blocking: no
>>>>> direct reclaim, no direct compaction and no drain_all_pages() IPI
>>>>> across every CPU. kswapd (and in turn kcompactd) is still woken further
>>>>> down for background defragmentation, so compaction keeps working for
>>>>> long-term system health while being removed from the latency-critical
>>>>> direct allocation path.
>>>>> 
>>>>> Allocations that also request __GFP_THISNODE are exempted.  That flag
>>>>> pairing identifies the local-node-first THP attempt issued by
>>>>> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
>>>>> form transparent huge pages.
>>>>> 
>>>>> Test environment:
>>>>>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>>>>>              12x 1TB IO2 32000 IOPS RAID0 XFS
>>>>>   OS:        AL2023
>>>>>   Kernel:    v7.3-rc1
>>>>>   Database:  PostgreSQL 18.4
>>>>>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
>>>>> 
>>>>> Results (average of 3 runs, TPS):
>>>>> 
>>>>>   Config                   Avg TPS      % vs Baseline
>>>>>   baseline (no patch)       59,408       -
>>>>>   With this patch          155,409      +161.6%
>>>>> 
>>>>> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@xxxxxxxxx/T/#t [v1]
>>>>> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@xxxxxxxxx/T/#u [v2]
>>>>> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@xxxxxxxxx/T/#u [v3]
>>>>> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>>>> Cc: stable@xxxxxxxxxxxxxxx
>>>>> Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>>>>> Cc: Vlastimil Babka <vbabka@xxxxxxx>
>>>>> Cc: David Hildenbrand <david@xxxxxxxxxx>
>>>>> Cc: Michal Hocko <mhocko@xxxxxxxx>
>>>>> Cc: Johannes Weiner <hannes@xxxxxxxxxxx>
>>>>> Cc: Matthew Wilcox <willy@xxxxxxxxxxxxx>
>>>>> Cc: Christoph Hellwig <hch@xxxxxx>
>>>>> Cc: Dave Chinner <dgc@xxxxxxxxxx>
>>>>> Cc: Ritesh Harjani <ritesh.list@xxxxxxxxx>
>>>>> Cc: linux-mm@xxxxxxxxx
>>>>> Cc: linux-fsdevel@xxxxxxxxxxxxxxx
>>>>> Cc: linux-xfs@xxxxxxxxxxxxxxx
>>>>> Signed-off-by: Salvatore Dipietro <dipiets@xxxxxxxxx>
>>>>
>>>> I guess this will have to do until unlikely(we figure out a better API)...
>>> 
>>> We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on
>>> various policies we currently have.
>>
>> Maybe with the help of alloc_flags to avoid the limitated count of gfp flags.
> 
> But alloc_flags are not exposed outside of MM, so alloc_page*() users
> are still limited by the existing GFP_* flags. Are you suggesting
> alloc_flags could be an addtional input for alloc_page*()?
> 
>>
>>> Even better if callers can do that
>>> instead.
>>
>> Not sure about burdening the callers (e.g. "every filesystem should do X"
>> etc), unless they are some intermediate wrappers within mm itself.
> 
> Not all callers. I assume gfp_policy() will only be used by advance
> users who want specific page allocation behavior without knowing the
> details of the page allocator or changing how the page allocator behave.

"without knowing the details"

>> For example, THP (but only anonymous?) is now handled in such a special way,
>> it could be a candidate. But I also have a vague feeling this was already
>> done in the past. Worth investigating perhaps.
>>
> 
> The number of wrappers can grow if different users have different needs.
> Some might not want direct reclaim, some might not want direct
> compaction, some might not want kswapd, and so on. I am not sure we want
> to add wrappers for each.

Yet all these are the details?

> For this patch, the caller can use gfp_policy() to strip out
> __GFP_DIRECT_RECLAIM based on the conditions and pass the adjusted gfp
> to the page allocator.

I guess I'd need to see a concrete example to grasp this fully.