[PATCH 2/2] mm: nommu: fix error handling on vma_iter_prealloc() failure

[Hajime Tazaki]

If do_mmap() finds an existing overlapping shared region, it increments
its usage, sets region to this existing pregion, and jumps to share:

	region = pregion;
	result = start;
	goto share;

When vma_iter_prealloc() fails and jumps to error_vma_iter_prealloc, the
error path unconditionally frees the region:

error:
	...
	if (region->vm_file)
		fput(region->vm_file);
	kmem_cache_free(vm_region_jar, region);

This potentially leaves a dangling pointer in nommu_region_tree and
causes RB-tree corruption.

Additionally, when establishing a new private mapping, do_mmap_private()
allocates physical pages and assigns them to region->vm_start:

	base = alloc_pages_exact(total << PAGE_SHIFT, GFP_KERNEL);
	...
	region->vm_start = (unsigned long) base;

If we later fail at vma_iter_prealloc() and jump to
error_vma_iter_prealloc, the region struct is freed, but the backing
physical memory doesn't seem to be freed via free_page_series().

This commit fixes those issues, discovered Sashiko, linked below.

Link: https://sashiko.dev/#/patchset/c8513ee5aa8444ec9bf6c276043c9f833016a2fa.1783304131.git.thehajime%40gmail.com
Signed-off-by: Hajime Tazaki <thehajime@xxxxxxxxx>
---
 mm/nommu.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/mm/nommu.c b/mm/nommu.c
index ad619611e83f..24bd8c529c2a 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1222,9 +1222,16 @@ unsigned long do_mmap(struct file *file,
 	up_write(&nommu_region_sem);
 error:
 	vma_iter_free(&vmi);
-	if (region->vm_file)
-		fput(region->vm_file);
-	kmem_cache_free(vm_region_jar, region);
+
+	/* if the error was from shared mapping/existing region, don't free the region  */
+	if (region->vm_usage == 1) {
+		if (region->vm_file)
+			fput(region->vm_file);
+		kmem_cache_free(vm_region_jar, region);
+
+	} else
+		region->vm_usage--;
+
 	if (vma->vm_file)
 		fput(vma->vm_file);
 	vm_area_free(vma);
@@ -1240,6 +1247,11 @@ unsigned long do_mmap(struct file *file,
 	up_write(&nommu_region_sem);
 	pr_warn("Failed vma_iter_prealloc()\n");
 	ret = -ENOMEM;
+
+	/* in case that the region is allocated via do_mmap_private() */
+	if ((region->vm_usage == 1) &&
+	    (!file || !(vma->vm_flags & VM_SHARED)))
+		free_page_series(region->vm_start, region->vm_top);
 	goto error;
 
 error_getting_vma:
-- 
2.43.0