[PATCH] hugetlb: retry private VMA locking when unmapping folios

[Daehyeon Ko]

hugetlb_unmap_file_folio() takes a reference to the hugetlb_vma_lock in
vm_private_data when its VMA trylock fails. It then drops i_mmap_rwsem and
waits for the lock in the established order.

That fallback is only valid for VM_MAYSHARE VMAs. Since commit
bf4916922c60 ("hugetlbfs: extend hugetlb_vma_lock to private VMAs"), the
trylock can also fail for a private owner VMA. Its vm_private_data contains
resv_map | HPAGE_RESV_OWNER, not a hugetlb_vma_lock. The fallback then
performs a misaligned refcount operation and treats reservation-map list
fields as an rwsem.

Private reservation maps have no stable VMA backpointer. Drop i_mmap_rwsem
and restart the interval-tree walk when their trylock fails. Keep the
referenced fallback for shareable VMAs.

On a system with a pre-provisioned hugepage pool, an unprivileged
reproducer racing mprotect() with hole punching triggered the refcount
warning followed by a general protection fault in rwsem_down_write_slowpath
on v7.3-rc5 and 6.12.111. With this change, a v7.3-rc5 KASAN+LOCKDEP build
completed 1,000 rounds and 64,000 hole punches without a warning, lockdep
report, oops, or panic. The reproducer is available privately on request.

Fixes: bf4916922c60 ("hugetlbfs: extend hugetlb_vma_lock to private VMAs")
Cc: stable@xxxxxxxxxxxxxxx
Assisted-by: LLM
Signed-off-by: Daehyeon Ko <4ncienth@xxxxxxxxx>
---
 fs/hugetlbfs/inode.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 7611a8470ea265..9a676228e2070a 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -402,6 +402,17 @@ static void hugetlb_unmap_file_folio(struct hstate *h,
 			continue;
 
 		if (!hugetlb_vma_trylock_write(vma)) {
+			/*
+			 * The referenced fallback below is specific to the
+			 * hugetlb_vma_lock stored by shareable VMAs.  A private
+			 * owner VMA stores a tagged resv_map pointer here instead.
+			 */
+			if (!(vma->vm_flags & VM_MAYSHARE)) {
+				i_mmap_unlock_write(mapping);
+				cond_resched();
+				i_mmap_lock_write(mapping);
+				goto retry;
+			}
 			vma_lock = vma->vm_private_data;
 			/*
 			 * If we can not get vma lock, we need to drop

base-commit: 0fcb05b72344f40806cbe810de8872f1698bf053