Re: [RFC PATCH v4 1/3] mm: allow page faults to request VMA-lock retry
From: Hongru Zhang
Date: Sat Aug 08 2026 - 06:58:55 EST
> Swap Throughput (higher is better):
> +--------------+-------------+----------------------+----------------------+
> | mmap writers | Vanilla | P1 | P3 |
> +--------------+-------------+----------------------+----------------------+
> | 0 | 17303.09 /s | 17899.48 /s (+3.4%) | 18190.62 /s (+5.1%) |
> +--------------+-------------+----------------------+----------------------+
> | 2 | 16728.04 /s | 18346.72 /s (+9.7%) | 18162.03 /s (+8.6%) |
> +--------------+-------------+----------------------+----------------------+
> | 4 | 12596.23 /s | 16095.20 /s (+27.8%) | 17991.45 /s (+42.8%) |
> +--------------+-------------+----------------------+----------------------+
>
> P3:
>
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index 45b99c3b1442..c3ab30d32a15 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1222,6 +1222,7 @@ void do_user_addr_fault(struct pt_regs *regs,
> struct mm_struct *mm;
> vm_fault_t fault;
> unsigned int flags = FAULT_FLAG_DEFAULT;
> + bool vma_lock_retried = false;
>
> tsk = current;
> mm = tsk->mm;
> @@ -1331,6 +1332,7 @@ void do_user_addr_fault(struct pt_regs *regs,
> if (!(flags & FAULT_FLAG_USER))
> goto lock_mmap;
>
> +lock_vma:
> vma = lock_vma_under_rcu(mm, address);
> if (!vma)
> goto lock_mmap;
> @@ -1360,6 +1362,12 @@ void do_user_addr_fault(struct pt_regs *regs,
> ARCH_DEFAULT_PKEY);
> return;
> }
> +
> + if (!vma_lock_retried) {
> + vma_lock_retried = true;
> + goto lock_vma;
> + }
> +
> lock_mmap:
>
> retry:
> diff --git a/mm/memory.c b/mm/memory.c
> index 428eb555ecb7..8c34a857548b 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -4987,6 +4987,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
> }
>
> swapcache = folio;
> + /*
> + * If the folio is uptodate, we are likely only waiting for
> + * another concurrent PTE mapping to complete, which should
> + * be brief. No need to drop the lock and retry the fault.
> + */
> + if (folio_test_uptodate(folio))
> + vmf->flags &= ~FAULT_FLAG_ALLOW_RETRY;
> ret |= folio_lock_or_retry(folio, vmf);
> if (ret & VM_FAULT_RETRY)
> goto out_release;
>
> Thanks,
> Hongru
The two-writer results varied too much to show a clear difference, so I
increased the number of writers to eight.
Using a dedicated benchmark [1], we tested this on a 20-core Intel
i7-12700 desktop with a 2GB swapfile. The benchmark uses one pressure
thread under memcg limits to keep a 128MB non-zero anonymous mapping
under swap pressure, 12 reader threads to fault it back in, and optional
mmap writer threads to amplify mmap_lock read-write contention. Each
test ran for 60 seconds and reports completed reader rounds per second
under swap pressure.
Swap Throughput (higher is better):
+--------------+-------------+---------------------------+---------------------------+
| mmap writers | Vanilla | P1 | P3 |
+--------------+-------------+---------------------------+---------------------------+
| 0 | 17303.09 /s | 17899.48 /s (+3.4%) | 18190.62 /s (+5.1%) |
+--------------+-------------+---------------------------+---------------------------+
| 4 | 12596.23 /s | 16095.20 /s (+27.8%) | 17991.45 /s (+42.8%) |
+--------------+-------------+---------------------------+---------------------------+
| 8 | 0.58 /s | 15420.57 /s (+2658619.0%) | 17328.80 /s (+2987624.1%) |
+--------------+-------------+---------------------------+---------------------------+
With increasing mmap_lock write pressure, Vanilla degrades sharply and
drops to near zero at eight writers. P1 and P3 hold up much better, with
P3 consistently outperforming P1.
P1 retries once under the VMA lock before falling back to mmap_lock. P3
additionally skips the retry in do_swap_page() when the folio is already
uptodate.
If there are no objections, I plan to respin with P1. The do_swap_page()
change will be sent as a separate patch.
P1:
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 45b99c3b1442..c3ab30d32a15 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1222,6 +1222,7 @@ void do_user_addr_fault(struct pt_regs *regs,
struct mm_struct *mm;
vm_fault_t fault;
unsigned int flags = FAULT_FLAG_DEFAULT;
+ bool vma_lock_retried = false;
tsk = current;
mm = tsk->mm;
@@ -1331,6 +1332,7 @@ void do_user_addr_fault(struct pt_regs *regs,
if (!(flags & FAULT_FLAG_USER))
goto lock_mmap;
+lock_vma:
vma = lock_vma_under_rcu(mm, address);
if (!vma)
goto lock_mmap;
@@ -1360,6 +1362,12 @@ void do_user_addr_fault(struct pt_regs *regs,
ARCH_DEFAULT_PKEY);
return;
}
+
+ if (!vma_lock_retried) {
+ vma_lock_retried = true;
+ goto lock_vma;
+ }
+
lock_mmap:
P3:
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 45b99c3b1442..c3ab30d32a15 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1222,6 +1222,7 @@ void do_user_addr_fault(struct pt_regs *regs,
struct mm_struct *mm;
vm_fault_t fault;
unsigned int flags = FAULT_FLAG_DEFAULT;
+ bool vma_lock_retried = false;
tsk = current;
mm = tsk->mm;
@@ -1331,6 +1332,7 @@ void do_user_addr_fault(struct pt_regs *regs,
if (!(flags & FAULT_FLAG_USER))
goto lock_mmap;
+lock_vma:
vma = lock_vma_under_rcu(mm, address);
if (!vma)
goto lock_mmap;
@@ -1360,6 +1362,12 @@ void do_user_addr_fault(struct pt_regs *regs,
ARCH_DEFAULT_PKEY);
return;
}
+
+ if (!vma_lock_retried) {
+ vma_lock_retried = true;
+ goto lock_vma;
+ }
+
lock_mmap:
retry:
diff --git a/mm/memory.c b/mm/memory.c
index 428eb555ecb7..8c34a857548b 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4987,6 +4987,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
}
swapcache = folio;
+ /*
+ * If the folio is uptodate, we are likely only waiting for
+ * another concurrent PTE mapping to complete, which should
+ * be brief. No need to drop the lock and retry the fault.
+ */
+ if (folio_test_uptodate(folio))
+ vmf->flags &= ~FAULT_FLAG_ALLOW_RETRY;
ret |= folio_lock_or_retry(folio, vmf);
if (ret & VM_FAULT_RETRY)
goto out_release;
[1] https://gist.github.com/zhr250/218ffe693f842346b56434483127422c
Thanks,
Hongru