Re: [PATCH v4] mm/page_alloc: avoid direct compaction for costly __GFP_NORETRY allocations

[Vlastimil Babka (SUSE)]

On 9/4/26 17:08, Zi Yan wrote:
> On Fri Sep 4, 2026 at 10:11 AM EDT, Vlastimil Babka (SUSE) wrote:
>> On 9/4/26 13:56, Salvatore Dipietro wrote:
>>> Commit 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>> introduced high-order folio allocations in the iomap buffered write
>>> path.  When memory is fragmented, each failed costly-order allocation
>>> enters __alloc_pages_slowpath() which runs direct compaction and
>>> drain_all_pages(), causing a 0.38x throughput drop on PostgreSQL
>>> pgbench (simple-update) with 1024 clients on a 96-vCPU arm64 system.
>>> 
>>> The root issue is that direct compaction is too expensive for hot
>>> allocation paths that have fallbacks to smaller allocations.
>>> __filemap_get_folio_mpol() already marks higher-order allocations with
>>> __GFP_NORETRY | __GFP_NOWARN, signalling that the caller can handle
>>> failure.  However, the page allocator still attempts full direct
>>> compaction for costly orders with __GFP_NORETRY, which is unnecessarily
>>> aggressive when the caller will simply retry at a lower order.
>>> 
>>> For costly-order allocations with __GFP_NORETRY, clear
>>> __GFP_DIRECT_RECLAIM at the very start of the slowpath, before
>>> can_direct_reclaim, can_compact and the nofail checks are evaluated.
>>> This makes the entire slowpath treat the request as non-blocking: no
>>> direct reclaim, no direct compaction and no drain_all_pages() IPI
>>> across every CPU. kswapd (and in turn kcompactd) is still woken further
>>> down for background defragmentation, so compaction keeps working for
>>> long-term system health while being removed from the latency-critical
>>> direct allocation path.
>>> 
>>> Allocations that also request __GFP_THISNODE are exempted.  That flag
>>> pairing identifies the local-node-first THP attempt issued by
>>> alloc_pages_mpol() (mempolicy.c), which relies on direct compaction to
>>> form transparent huge pages.
>>> 
>>> Test environment:
>>>   Hardware:  AWS EC2 m8g.24xlarge (96 vCPU, arm64)
>>>              12x 1TB IO2 32000 IOPS RAID0 XFS
>>>   OS:        AL2023
>>>   Kernel:    v7.3-rc1
>>>   Database:  PostgreSQL 18.4
>>>   Workload:  pgbench simple-update, 1024 clients, 96 threads, 1200s
>>> 
>>> Results (average of 3 runs, TPS):
>>> 
>>>   Config                   Avg TPS      % vs Baseline
>>>   baseline (no patch)       59,408       -
>>>   With this patch          155,409      +161.6%
>>> 
>>> Link: https://lore.kernel.org/all/20260403193535.9970-1-dipiets@xxxxxxxxx/T/#t [v1]
>>> Link: https://lore.kernel.org/linux-mm/20260420161404.642-1-dipiets@xxxxxxxxx/T/#u [v2]
>>> Link: https://lore.kernel.org/all/20260710143437.12379-1-dipiets@xxxxxxxxx/T/#u [v3]
>>> Fixes: 5d8edfb900d5 ("iomap: Copy larger chunks from userspace")
>>> Cc: stable@xxxxxxxxxxxxxxx
>>> Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>>> Cc: Vlastimil Babka <vbabka@xxxxxxx>
>>> Cc: David Hildenbrand <david@xxxxxxxxxx>
>>> Cc: Michal Hocko <mhocko@xxxxxxxx>
>>> Cc: Johannes Weiner <hannes@xxxxxxxxxxx>
>>> Cc: Matthew Wilcox <willy@xxxxxxxxxxxxx>
>>> Cc: Christoph Hellwig <hch@xxxxxx>
>>> Cc: Dave Chinner <dgc@xxxxxxxxxx>
>>> Cc: Ritesh Harjani <ritesh.list@xxxxxxxxx>
>>> Cc: linux-mm@xxxxxxxxx
>>> Cc: linux-fsdevel@xxxxxxxxxxxxxxx
>>> Cc: linux-xfs@xxxxxxxxxxxxxxx
>>> Signed-off-by: Salvatore Dipietro <dipiets@xxxxxxxxx>
>>
>> I guess this will have to do until unlikely(we figure out a better API)...
> 
> We could add a "new_gfp = gfp_policy(gfp)" to adjust input gfp based on
> various policies we currently have.

Maybe with the help of alloc_flags to avoid the limitated count of gfp flags.

> Even better if callers can do that
> instead.

Not sure about burdening the callers (e.g. "every filesystem should do X"
etc), unless they are some intermediate wrappers within mm itself.
For example, THP (but only anonymous?) is now handled in such a special way,
it could be a candidate. But I also have a vague feeling this was already
done in the past. Worth investigating perhaps.

>>
>> Acked-by: Vlastimil Babka (SUSE) <vbabka@xxxxxxxxxx>
>>
>>> ---
>>> v4: Clear __GFP_DIRECT_RECLAIM early in the slowpath and exempt
>>>     __GFP_THISNODE so THP attempt keeps using direct compaction
>>> v3: Move to mm/page_alloc.c, wake kcompactd instead of avoiding it
>>> v2: Move from fs/iomap/buffered-io.c to mm/filemap.c
>>> v1: Avoid compaction in iomap folio allocation
>>> 
>>>  mm/page_alloc.c | 18 +++++++++++++++---
>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>> 
>>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
>>> index 12fac9084c48..542c2ec31061 100644
>>> --- a/mm/page_alloc.c
>>> +++ b/mm/page_alloc.c
>>> @@ -4784,10 +4784,10 @@ static inline struct page *
>>>  __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>>  						struct alloc_context *ac)
>>>  {
>>> -	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>>> -	bool can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
>>> -	bool nofail = gfp_mask & __GFP_NOFAIL;
>>>  	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
>>> +	bool can_direct_reclaim;
>>> +	bool can_compact;
>>> +	bool nofail;
>>>  	struct page *page = NULL;
>>>  	unsigned int alloc_flags;
>>>  	unsigned long did_some_progress;
>>> @@ -4802,6 +4802,18 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>>>  	bool can_retry_reserves = true;
>>>  	unsigned long alloc_start_time = jiffies;
>>>  
>>> +	/*
>>> +	 * Costly __GFP_NORETRY callers have a cheap fallback, so don't stall
>>> +	 * them in reclaim or compaction. __GFP_THISNODE callers are exempt.
>>> +	 */
> 
> Should this also be documented in gfp_types.h? It currently only says,
> 
> "__GFP_NORETRY: The VM implementation will try only very lightweight
> memory direct reclaim to get some memory under memory pressure (thus it
> can sleep)."
> 
> Otherwise,
> 
> Acked-by: Zi Yan <ziy@xxxxxxxxxx>
> 
>>> +	if (costly_order && (gfp_mask & __GFP_NORETRY) &&
>>> +	    !(gfp_mask & __GFP_THISNODE))
>>> +		gfp_mask &= ~__GFP_DIRECT_RECLAIM;
>>> +
>>> +	can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
>>> +	can_compact = can_direct_reclaim && gfp_compaction_allowed(gfp_mask);
>>> +	nofail = gfp_mask & __GFP_NOFAIL;
>>> +
>>>  	if (unlikely(nofail)) {
>>>  		/*
>>>  		 * Also we don't support __GFP_NOFAIL without __GFP_DIRECT_RECLAIM,
> 
> 
> 
>