[PATCH 42/60] mm: fail private placement on exhaustion instead of OOM killing

From: Gregory Price via B4 Relay

Date: Fri Nov 20 2026 - 13:21:55 EST


From: "Gregory Price (Meta)" <gourry@xxxxxxxxxx>

Private memory belongs to a device or specialized service and is a fixed
pool, like the HugeTLB pool. The OOM killer has no good answer for it:
the task that faults the last page may not be the one that filled the
node, a remote accessor or another mapper of a shared object would be
blamed, and killing a task does not free shared private pages.

When reclaim cannot satisfy an allocation from a private zonelist, fail
it instead of OOMing.
- __alloc_pages_may_oom() does not select a victim for a private
request, and marks the task.
- handle_mm_fault() turns VM_FAULT_OOM from such a failure into
VM_FAULT_SIGBUS. GUP and remote access see -EFAULT, and other
callers see the allocation failure, e.g. -ENOMEM from write(2).

A request that became ineligible since the last retry check is not
exhausted. Report progress so the slowpath retries and relaxes it.

Signed-off-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
---
include/linux/sched.h | 2 ++
mm/memory.c | 5 +++++
mm/page_alloc.c | 14 ++++++++++++++
3 files changed, 21 insertions(+)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index e4f6323d7855..8a2e37475e5d 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1038,6 +1038,8 @@ struct task_struct {
/* whether the LRU algorithm may apply to this access */
unsigned in_lru_fault:1;
#endif
+ /* A private memory allocation failed; a fault reports SIGBUS. */
+ unsigned private_alloc_failed:1;
#ifdef CONFIG_COMPAT_BRK
unsigned brk_randomized:1;
#endif
diff --git a/mm/memory.c b/mm/memory.c
index 282266dd5e5d..4a8a08df28ff 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -7003,6 +7003,7 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
}

is_droppable = !!(vma->vm_flags & VM_DROPPABLE);
+ current->private_alloc_failed = 0;

/*
* Enable the memcg OOM handling for faults triggered in user
@@ -7034,6 +7035,10 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
if (is_droppable && (flags & FAULT_FLAG_USER))
ret &= ~VM_FAULT_OOM;

+ /* Exhausted private memory fails like an empty HugeTLB pool. */
+ if (unlikely(current->private_alloc_failed) && (ret & VM_FAULT_OOM))
+ ret = (ret & ~VM_FAULT_OOM) | VM_FAULT_SIGBUS;
+
if (flags & FAULT_FLAG_USER) {
mem_cgroup_exit_user_fault();
/*
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 63cc8729c41a..ea48991ef589 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4176,6 +4176,20 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
* failures more gracefully we should just bail out here.
*/

+ /*
+ * Private memory is a fixed pool, exhausting it fails the allocation
+ * rather than selecting an OOM victim. Otherwise, if there is some
+ * indication of progress because of a revoked request, just report
+ * some progress so the slowpath retries and relaxes it.
+ */
+ if (ac->alloc_flags & ALLOC_ZONELIST_PRIVATE) {
+ if (!alloc_private_reachable(gfp_mask, ac))
+ *did_some_progress = 1;
+ else
+ current->private_alloc_failed = 1;
+ goto out;
+ }
+
/* Exhausted what can be done so it's blame time */
if (out_of_memory(&oc) ||
WARN_ON_ONCE_GFP(gfp_mask & __GFP_NOFAIL, gfp_mask)) {

--
2.56.1