[PATCH v2 3/4] mm/memory: do not report OOM if insert_pfn() races a huge fault

From: Lorenzo Stoakes (ARM)

Date: Tue Nov 17 2026 - 13:33:36 EST


A driver that implements both ->fault and ->huge_fault must anticipate
concurrent use of both.

This can happen, for example, if a user calls prctl(PR_SET_THP_DISABLE)
concurrent with faulting, as that only holds the mmap write lock and thus
does not exclude faulting threads stabilised by VMA lock.

This is not currently safely handled by insert_pfn(), however, which
obtains its PTE entry via get_locked_pte(), and treats a NULL return value
as an allocation failure, returning VM_FAULT_OOM.

The issue is that get_locked_pte() also returns NULL if the PMD entry is
not a page table which can happen if a concurrent huge fault installs a
huge PMD entry between __handle_mm_fault() observing the PMD entry to be
empty and insert_pfn() being invoked.

This can result in VM_FAULT_OOM being reported incorrectly.

Fix this by introducing and using __get_locked_pte() which returns -ENOMEM
if a page table allocation fails or -EFAULT if raced.

This allows insert_pfn() to return VM_FAULT_OOM only when an allocation
fails and VM_FAULT_NOPAGE otherwise so the fault is retried.

The same change is applied to insert_page(), which only impacts DAX (which
has its own locking).

This means that insert_page() now propagates the returned error, so -EFAULT
in the case of a race. It's not possible currently, but guards against that
changing in future.

Similarly, update walk_to_pmd() to return -EFAULT if it encounters a huge
PUD leaf.

Target the commit introducing the first driver able to do this.

Reported-by: Eliav Farber <farbere@xxxxxxxxxx>
Closes: https://lore.kernel.org/all/20261112051440.16140-3-farbere@xxxxxxxxxx/
Fixes: f9e54c3a2f5b ("vfio/pci: implement huge_fault support")
Cc: stable@xxxxxxxxxxxxxxx
Suggested-by: Jose A. Perez de Azpillaga <azpijr@xxxxxxxxx>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
---
mm/memory.c | 48 ++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 40 insertions(+), 8 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 20b818bcdfbc..04a518d6059f 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2383,19 +2383,48 @@ static int walk_to_pmd(struct mm_struct *mm, unsigned long addr, pmd_t **pmdp)
pmd = pmd_alloc(mm, pud, addr);
if (!pmd)
return -ENOMEM;
+ /* We may have raced a huge PUD fault. */
+ if (pud_trans_unstable(pud))
+ return -EFAULT;

*pmdp = pmd;
return 0;
}

+/*
+ * Map and lock the PTE entry for addr, allocating page tables as necessary.
+ *
+ * Returns 0 on success, -ENOMEM if a page table could not be allocated, or
+ * -EFAULT if we raced and found something other than a page table.
+ */
+static int __get_locked_pte(struct mm_struct *mm, unsigned long addr,
+ pte_t **ptep, spinlock_t **ptl)
+{
+ pmd_t *pmd;
+ pte_t *pte;
+ int err;
+
+ err = walk_to_pmd(mm, addr, &pmd);
+ if (err)
+ return err;
+ if (pte_alloc(mm, pmd))
+ return -ENOMEM;
+ pte = pte_offset_map_lock(mm, pmd, addr, ptl);
+ if (!pte)
+ return -EFAULT;
+
+ *ptep = pte;
+ return 0;
+}
+
pte_t *get_locked_pte(struct mm_struct *mm, unsigned long addr,
spinlock_t **ptl)
{
- pmd_t *pmd;
+ pte_t *pte;

- if (walk_to_pmd(mm, addr, &pmd))
+ if (__get_locked_pte(mm, addr, &pte, ptl))
return NULL;
- return pte_alloc_map_lock(mm, pmd, addr, ptl);
+ return pte;
}

static bool vm_mixed_zeropage_allowed(struct vm_area_struct *vma)
@@ -2497,9 +2526,8 @@ static int insert_page(struct vm_area_struct *vma, unsigned long addr,
retval = validate_page_before_insert(vma, page);
if (retval)
goto out;
- retval = -ENOMEM;
- pte = get_locked_pte(vma->vm_mm, addr, &ptl);
- if (!pte)
+ retval = __get_locked_pte(vma->vm_mm, addr, &pte, &ptl);
+ if (retval)
goto out;
retval = insert_page_into_pte_locked(vma, pte, addr, page, prot,
mkwrite);
@@ -2865,10 +2893,14 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr,
struct mm_struct *mm = vma->vm_mm;
pte_t *pte, entry;
spinlock_t *ptl;
+ int err;

- pte = get_locked_pte(mm, addr, &ptl);
- if (!pte)
+ err = __get_locked_pte(mm, addr, &pte, &ptl);
+ if (err == -ENOMEM)
return VM_FAULT_OOM;
+ /* We may have raced a huge fault, if so, retry the fault. */
+ if (err)
+ return VM_FAULT_NOPAGE;
entry = ptep_get(pte);
if (!pte_none(entry)) {
if (mkwrite) {

--
2.56.0