Re: [PATCH v2 3/5] mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

[Joshua Hahn]

On Wed, 08 Jul 2026 15:12:51 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@xxxxxxxxxx> wrote:

Hi Ackerley, 

Thanks again for the fix. Thank you also for adding the reproducers,
they made testing this really easy for me! : -D

> From: Ackerley Tng <ackerleytng@xxxxxxxxxx>
> 
> When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has
> its refcount set to 1 via folio_ref_unfreeze(folio, 1).
> 
> The error path calls free_huge_folio(folio) directly, which expects a
> refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is
> triggered.

So yeah, I built mm-new (with your hugetlb open code series) and ran
reproducer #3, and got the following output:

...
[   23.855813] Call Trace:
[   23.855838]  <TASK>
[   23.855877]  hugetlb_alloc_folio+0x190/0x360
[   23.855930]  alloc_hugetlb_folio+0xf9/0x310
[   23.855972]  hugetlb_no_page+0x590/0xa00
[   23.856016]  hugetlb_fault+0x194/0x770
[   23.856060]  handle_mm_fault+0x29b/0x2c0
[   23.856103]  do_user_addr_fault+0x208/0x6e0
[   23.856146]  exc_page_fault+0x67/0x140
[   23.856191]  asm_exc_page_fault+0x22/0x30
[   23.856234] RIP: 0033:0x401734
...

Running it with your fix, I don't get a kernel crash. So this change
looks good to me, please feel free to add:

Tested-by: Joshua Hahn <joshua.hahnjy@xxxxxxxxx>
Reviewed-by: Joshua Hahn <joshua.hahnjy@xxxxxxxxx>

But.....

It seems like fixing this bug actually surfaces a different bug.
I don't see a kernel crash anymore, but my kernel gets stuck in a 
different loop:

...
[   42.526726] pagefault_out_of_memory: 120780 callbacks suppressed
[   42.526732] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.526955] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527084] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527192] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527288] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527382] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
[   42.527476] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
...

mem_cgroup_charge_hugetlb() fails
--> hugetlb_alloc_folio() returns ERR_PTR(-ENOMEM)
--> propagated to alloc_hugetlb_folio()
--> hugetlb_no_page converts it to vmf_error(PTR_ERROR(folio))
--> propagates up to the pagefault handler..

But there's no OOM handler to resolve, I think. So it just keeps retrying
the fault and cycling over and over and over again... I think we need
to be returning ENOSPC instead of ENOMEM like the other failure paths
in alloc_hugetlb_folio (or hugetlb_alloc_folio... I think we should
change this name, it's a bit confusing).

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 288535838a48b..05214b1cd491a 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2903,7 +2903,7 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
                 * were committed to the folio and freeing the folio
                 * would have cleared those up.
                 */
-               return ERR_PTR(ret);
+               return ERR_PTR(-ENOSPC);
        }

        return folio;

This gives me the following result from your reproducer, which I think
is the expected behavior:

attempting to remount cgroup2 with memory_hugetlb_accounting...
Successfully enabled memory_hugetlb_accounting
Child: Attempting to allocate and touch 2MB hugepage...
Child: mmap succeeded at 0x7f0023600000, touching it now (should trigger fault)...
Parent: Child exited. Cleaning up.
Parent: Child killed by signal 7 (Bus error)
Parent: Child got SIGBUS as expected (if kernel didn't crash).

What do you think? Happy to send it out as a separate fix or feel free
to fold it into your series / fix, whatever you prefer.

Thanks Ackerley!

> Even with CONFIG_DEBUG_VM disabled, returning a folio with refcount 1 to
> the freelist can corrupt allocator state later.
> 
> Use folio_put(folio) instead of free_huge_folio(folio) to properly drop the
> reference before freeing it.
> Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
> Cc: stable@xxxxxxxxxxxxxxx
> Signed-off-by: Ackerley Tng <ackerleytng@xxxxxxxxxx>
> Reviewed-by: Muchun Song <muchun.song@xxxxxxxxx>
> ---
>  mm/hugetlb.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 4093c1c0a4a1d..1f3f4b964b153 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -2990,7 +2990,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
>  	lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
>  
>  	if (ret == -ENOMEM) {
> -		free_huge_folio(folio);
> +		folio_put(folio);
>  		return ERR_PTR(-ENOMEM);
>  	}
>  
> 
> -- 
> 2.55.0.795.g602f6c329a-goog