Re: [PATCH] mm/hugetlb: do not invoke the OOM killer when optimizing vmemmap

[Muchun Song]

> On Nov 4, 2026, at 21:39, Yili Zhang <zhangyili01@xxxxxxxxx> wrote:
> 
> Boot-time hugepage allocation is designed to drain all free memory and
> let the vmemmap optimization reclaim vmemmap pages, so that more
> hugepages can be allocated afterwards:
> hugetlb_pages_alloc_boot_node() flushes the folios accumulated so far
> once si_mem_available() drops to zero, and hugetlb_pages_alloc_boot()
> keeps retrying as long as progress is being made.  The allocation is
> parallelized with padata and runs on many threads at once.
> 
> The page that will back the new head struct page of a folio being
> optimized is allocated with a bare GFP_KERNEL mask in
> __hugetlb_vmemmap_optimize_folio():
> 
> vmemmap_head = alloc_pages_node(nid, GFP_KERNEL, 0);
> 
> Once the workers have consumed all free memory, this allocation enters
> the OOM path.  During boot-time allocation, at the do_one_initcall()
> stage, there are no user processes yet, so out_of_memory() finds
> nothing to kill and panics:
> 
>  Out of memory and no killable processes...
>  Kernel panic - not syncing: System is deadlocked on memory
>  CPU: 74 UID: 0 PID: 1 Comm: swapper/0
>  Call Trace:
>    vpanic+0x32f/0x360
>    panic+0x57/0x60
>    out_of_memory+0x581/0x590
>    __alloc_pages_slowpath.constprop.115+0x97d/0xed0
>    __alloc_frozen_pages_noprof+0x303/0x330
>    __alloc_pages_noprof+0xa/0x20
>    __hugetlb_vmemmap_optimize_folio+0xa3/0x210
>    __hugetlb_vmemmap_optimize_folios+0x190/0x230
>    prep_and_add_allocated_folios+0x16/0x90
>    hugetlb_pages_alloc_boot_node+0x152/0x180
>    padata_mt_helper+0x5b/0xb0
>    padata_do_multithreaded+0x294/0x3a0
>    hugetlb_hstate_alloc_pages+0x2e1/0x3b0
>    hugetlb_init+0x3b5/0x640
>    do_one_initcall+0x49/0x200
> 
> Optimizing the vmemmap is best-effort by design: on allocation failure,
> __hugetlb_vmemmap_optimize_folios() frees the vmemmap pages reclaimed
> from the folios it has processed so far and retries once, and folios
> that still cannot be optimized are put into the pool unoptimized by
> prep_and_add_allocated_folios().  The worst outcome of a failure is a
> higher vmemmap overhead and, as a consequence, fewer hugepages being
> allocated (along with a warning) - waking up the OOM killer is never
> the right thing to do here.  Note that the same code also runs for
> runtime hugepage allocations, where an OOM invocation may kill an
> unrelated process instead of failing the optimization gracefully.
> 
> The head page is a vmemmap page, and every other vmemmap page
> allocation involved is prepared to fail gracefully:
> vmemmap_alloc_block() - which vmemmap_shared_tail_page() also uses
> to allocate the shared tail vmemmap pages - uses
> GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN, and
> alloc_vmemmap_page_list() uses GFP_KERNEL|__GFP_RETRY_MAYFAIL.
> 
> Align it with the other vmemmap page allocations by using the same mask
> that vmemmap_alloc_block() uses for vmemmap pages, so that a failure
> returns -ENOMEM to the caller instead of going through the OOM path.
> 
> This was observed on a 384-CPU machine with ~1.5TiB of RAM booting
> with a hugepages= reservation close to the total memory size and
> CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y, where the boot
> panicked intermittently, depending on how the parallel allocation
> workers interleaved.

A really detailed analysis, but also lengthy (I think it's generated from
AI, It really should be trimmed before sending because the issue is very
obvious and easy to understand).

> 
> Fixes: c0b495b91a47b ("mm/hugetlb: refactor code around vmemmap_walk")
> Signed-off-by: Yili Zhang <zhangyili01@xxxxxxxxx>
> ---
> mm/hugetlb_vmemmap.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
> index 0057fa2a16a8..3a759b930ed9 100644
> --- a/mm/hugetlb_vmemmap.c
> +++ b/mm/hugetlb_vmemmap.c
> @@ -499,6 +499,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
> {
> 	unsigned long vmemmap_start, vmemmap_end;
> 	struct page *vmemmap_head, *vmemmap_tail;
> +	gfp_t gfp_mask = GFP_KERNEL | __GFP_NOWARN | __GFP_RETRY_MAYFAIL;

Thanks for fixing, but I realized that I've already fix it here [1], and I didn't
realize it's a real production issue. If you don't mind, I could add your Reported-by,
and make it a very small fix patch so that it can be backported more easily.

[1] https://lore.kernel.org/all/20261104023510.260766-5-songmuchun@xxxxxxxxxxxxx/

Thanks,
Muchun

> 	int nid, ret = 0;
> 
> 	VM_WARN_ON_ONCE_FOLIO(!folio_test_hugetlb(folio), folio);
> @@ -525,7 +526,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h,
> */
> 	folio_set_hugetlb_vmemmap_optimized(folio);
> 
> -	vmemmap_head = alloc_pages_node(nid, GFP_KERNEL, 0);
> +	vmemmap_head = alloc_pages_node(nid, gfp_mask, 0);
> 	if (!vmemmap_head) {
> 		ret = -ENOMEM;
> 		goto out;
> -- 
> 2.27.0
>