[PATCH 2/3] mm/memory: do not report OOM if insert_pfn() races a huge fault

From: Lorenzo Stoakes (ARM)

Date: Sun Nov 15 2026 - 16:00:07 EST


From: Eliav Farber <farbere@xxxxxxxxxx>

A driver that implements both ->fault and ->huge_fault must anticipate
concurrent use of both.

This can happen, for example, if a user calls prctl(PR_SET_THP_DISABLE)
concurrent with faulting, as that only holds the mmap write lock and thus
does not exclude faulting threads stabilised by VMA lock.

This is not currently safely handled by insert_pfn(), however, which
obtains its PTE entry via get_locked_pte(), and treats a NULL return value
as an allocation failure, returning VM_FAULT_OOM.

The issue is that get_locked_pte() also returns NULL if the PMD entry is
not a page table which can happen if a concurrent huge fault installs a
huge PMD entry between __handle_mm_fault() observing the PMD entry to be
empty and insert_pfn() being invoked.

This can result in VM_FAULT_OOM being reported incorrectly.

Address this by proactively checking whether a page table allocation
actually failed, and returning VM_FAULT_NOPAGE if the NULL indicates that
we were raced instead.

The same assumption is made in insert_page(), however only DAX implements
->huge_fault and can reach it, and it serialises faults of different orders
with its own mapping entry lock, so it does not require a change.

Target the commit introducing the first driver able to do this.

Link: https://lore.kernel.org/all/20261112051440.16140-3-farbere@xxxxxxxxxx/
Fixes: f9e54c3a2f5b ("vfio/pci: implement huge_fault support")
Cc: stable@xxxxxxxxxxxxxxx
Suggested-by: Jose A. Perez de Azpillaga <azpijr@xxxxxxxxx>
Signed-off-by: Eliav Farber <farbere@xxxxxxxxxx>
[ ljs: rewrote commit message, trimmed comment ]
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
---
mm/memory.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index e03110906aa8..cff39eccfd42 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2866,10 +2866,17 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr,
struct mm_struct *mm = vma->vm_mm;
pte_t *pte, entry;
spinlock_t *ptl;
+ pmd_t *pmd;

- pte = get_locked_pte(mm, addr, &ptl);
- if (!pte)
+ pmd = walk_to_pmd(mm, addr);
+ if (!pmd)
return VM_FAULT_OOM;
+ if (pte_alloc(mm, pmd))
+ return VM_FAULT_OOM;
+ /* We may have raced a huge fault, if so let the fault be retried. */
+ pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
+ if (!pte)
+ return VM_FAULT_NOPAGE;
entry = ptep_get(pte);
if (!pte_none(entry)) {
if (mkwrite) {

--
2.56.1