[RFC PATCH] mm/hugetlb: fix resv_map memory leak in __mmap_region error path

[Mingyu Wang]

While fuzzing with Syzkaller and fault injection (failslab) enabled,
I observed a persistent resv_map memory leak in the hugetlb mmap error path.

BUG: memory leak
unreferenced object 0xffff888110b92400 (size 512):
  comm "syz.0.5386", pid 20390, jiffies 4298157188
  backtrace:
    __kmalloc_cache_noprof+0x509/0x6e0
    resv_map_alloc+0x47/0x3a0
    hugetlb_reserve_pages+0x758/0x1220
    hugetlbfs_file_mmap_prepare+0x492/0x790
    __mmap_region+0x1ae6/0x29f0

This is a regression introduced by the recent VMA iterator and mmap region
refactoring, which decoupled mmap preparation from VMA completion.

In `__mmap_region()`, `call_mmap_prepare()` triggers `hugetlbfs_file_mmap_prepare()`,
which successfully allocates the `resv_map` and registers a `success_hook`
in `desc->action`.

If `__mmap_new_vma()` subsequently fails (e.g., `vma_iter_prealloc()`
returns -ENOMEM due to failslab), the code jumps to `abort_munmap`.
However, the `desc` structure is completely discarded without invoking
any cleanup. The newly allocated empty VMA is freed, but since
`set_vma_user_defined_fields()` was never reached, `vm_area_free()`
doesn't call `hugetlb_vm_close()`. Thus, the `resv_map` is permanently leaked.

This RFC proposes adding an `abort_hook` to `struct mmap_action`
so that subsystems can properly clean up resources allocated during the
`mmap_prepare` phase if VMA creation fails.

Any feedback on whether this architectural approach is correct, or how to 
properly implement the hugetlb unreserve rollback, would be highly appreciated.

Signed-off-by: Mingyu Wang <25181214217@xxxxxxxxxxxxxxxxx>
---
 fs/hugetlbfs/inode.c     | 9 +++++++++
 include/linux/mm_types.h | 2 ++
 mm/vma.c                 | 4 ++++
 3 files changed, 15 insertions(+)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 8b05bec08e04..002bb6d9ca23 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -102,6 +102,14 @@ static int hugetlb_file_mmap_prepare_success(const struct vm_area_struct *vma)
 	return hugetlb_vma_lock_alloc((struct vm_area_struct *)vma);
 }
 
+static void hugetlb_file_mmap_prepare_abort(struct vm_area_desc *desc)
+{
+	/*
+	 * TODO: Implement the proper rollback for hugetlb_reserve_pages()
+	 * and drop the resv_map reference held in the desc here.
+	 */
+}
+
 static int hugetlbfs_file_mmap_prepare(struct vm_area_desc *desc)
 {
 	struct file *file = desc->file;
@@ -172,6 +180,7 @@ static int hugetlbfs_file_mmap_prepare(struct vm_area_desc *desc)
 	if (!ret) {
 		/* Allocate the VMA lock after we set it up. */
 		desc->action.success_hook = hugetlb_file_mmap_prepare_success;
+		desc->action.abort_hook = hugetlb_file_mmap_prepare_abort;
 		/*
 		 * We cannot permit the rmap finding this VMA in the time
 		 * between the VMA being inserted into the VMA tree and the
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index a308e2c23b82..9320f6699fa9 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -861,6 +861,8 @@ struct mmap_action {
 	 * it is not valid to clear the error here.
 	 */
 	int (*error_hook)(int err);
+	
+	void (*abort_hook)(struct vm_area_desc *desc);
 
 	/*
 	 * This should be set in rare instances where the operation required
diff --git a/mm/vma.c b/mm/vma.c
index 377321b48734..d64cea5b4335 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -2799,6 +2799,10 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr,
 	 */
 	if (map.file_doesnt_need_get)
 		fput(map.file);
+	
+	if (have_mmap_prepare && desc.action.abort_hook)
+		desc.action.abort_hook(&desc);
+	
 	vms_abort_munmap_vmas(&map.vms, &map.mas_detach);
 	return error;
 }
-- 
2.34.1