Re: [PATCH v2 3/5] mm: hugetlb: Fix folio refcount mismatch on memcg charge failure

[Joshua Hahn]

On Thu,  9 Jul 2026 11:54:55 -0700 Joshua Hahn <joshua.hahnjy@xxxxxxxxx> wrote:

> On Wed, 08 Jul 2026 15:12:51 -0700 Ackerley Tng via B4 Relay <devnull+ackerleytng.google.com@xxxxxxxxxx> wrote:
> 
> Hi Ackerley, 
> 
> Thanks again for the fix. Thank you also for adding the reproducers,
> they made testing this really easy for me! : -D
> 
> > From: Ackerley Tng <ackerleytng@xxxxxxxxxx>
> > 
> > When mem_cgroup_charge_hugetlb(folio, gfp) returns -ENOMEM, the folio has
> > its refcount set to 1 via folio_ref_unfreeze(folio, 1).
> > 
> > The error path calls free_huge_folio(folio) directly, which expects a
> > refcount of 0. Hence, VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) is
> > triggered.
> 
> So yeah, I built mm-new (with your hugetlb open code series) and ran
> reproducer #3, and got the following output:
> 
> ...
> [   23.855813] Call Trace:
> [   23.855838]  <TASK>
> [   23.855877]  hugetlb_alloc_folio+0x190/0x360
> [   23.855930]  alloc_hugetlb_folio+0xf9/0x310
> [   23.855972]  hugetlb_no_page+0x590/0xa00
> [   23.856016]  hugetlb_fault+0x194/0x770
> [   23.856060]  handle_mm_fault+0x29b/0x2c0
> [   23.856103]  do_user_addr_fault+0x208/0x6e0
> [   23.856146]  exc_page_fault+0x67/0x140
> [   23.856191]  asm_exc_page_fault+0x22/0x30
> [   23.856234] RIP: 0033:0x401734
> ...
> 
> Running it with your fix, I don't get a kernel crash. So this change
> looks good to me, please feel free to add:
> 
> Tested-by: Joshua Hahn <joshua.hahnjy@xxxxxxxxx>
> Reviewed-by: Joshua Hahn <joshua.hahnjy@xxxxxxxxx>
> 
> But.....
> 
> It seems like fixing this bug actually surfaces a different bug.
> I don't see a kernel crash anymore, but my kernel gets stuck in a 
> different loop:
> 
> ...
> [   42.526726] pagefault_out_of_memory: 120780 callbacks suppressed
> [   42.526732] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.526955] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527084] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527192] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527288] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527382] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> [   42.527476] Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF
> ...
> 
> mem_cgroup_charge_hugetlb() fails
> --> hugetlb_alloc_folio() returns ERR_PTR(-ENOMEM)
> --> propagated to alloc_hugetlb_folio()
> --> hugetlb_no_page converts it to vmf_error(PTR_ERROR(folio))
> --> propagates up to the pagefault handler..
> 
> But there's no OOM handler to resolve, I think. So it just keeps retrying
> the fault and cycling over and over and over again... I think we need
> to be returning ENOSPC instead of ENOMEM like the other failure paths
> in alloc_hugetlb_folio (or hugetlb_alloc_folio... I think we should
> change this name, it's a bit confusing).
> 
> diff --git a/mm/hugetlb.c b/mm/hugetlb.c
> index 288535838a48b..05214b1cd491a 100644
> --- a/mm/hugetlb.c
> +++ b/mm/hugetlb.c
> @@ -2903,7 +2903,7 @@ struct folio *hugetlb_alloc_folio(struct hstate *h,
>                  * were committed to the folio and freeing the folio
>                  * would have cleared those up.
>                  */
> -               return ERR_PTR(ret);
> +               return ERR_PTR(-ENOSPC);
>         }
> 
>         return folio;

Ah, immediately as I sent this out and went to 4/5 to review it, I
see that this is the exact change you have... oops, please disregard
the message : -) I do think we should 3/5 and 4/5 together for
bisectability purposes.

Thanks again. Sorry for the noise!
Joshua