[PATCH bpf-next v7 1/4] bpf: Add a sleepable page allocator for map memory

[Jiayuan Chen]

bpf_map_alloc_pages() can use a non-blocking allocator that never
reclaims. Add bpf_map_alloc_page_sleepable() which always uses the
blocking allocator, so the allocation can reclaim. It uses
__GFP_RETRY_MAYFAIL so it never invokes the OOM killer: the page is
charged to the map's memcg, which need not be the caller's, so an OOM
there could kill unrelated tasks in the map's cgroup while a foreign
caller could never be its victim. The caller is responsible for a
context where blocking is safe; see the function comment.

Like the other bpf map allocators it places the page on the map's
numa_node and does not follow the faulting task's NUMA mempolicy; arena
memory is shared, so the map's node is the right placement policy. The
next patch uses it from the arena page fault handler.

Signed-off-by: Jiayuan Chen <jiayuan.chen@xxxxxxxxx>
Reviewed-by: Emil Tsalapatis <emil@xxxxxxxxxxxxxxx>
---
To sashiko:
- The alloc_pages_nolock() fallback still zeroes: it forces __GFP_ZERO
  internally and only accepts __GFP_ACCOUNT.
- __GFP_ZERO is unchanged from the existing __bpf_alloc_page(), and no
  arena-capable arch has D-cache aliasing, so there is no dcache concern.
---
 include/linux/bpf.h  |  1 +
 kernel/bpf/syscall.c | 28 ++++++++++++++++++++++++----
 2 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index e80963971f680..345fe4c2e6422 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2786,6 +2786,7 @@ struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);
 
 int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
 			unsigned long nr_pages, struct page **page_array);
+struct page *bpf_map_alloc_page_sleepable(const struct bpf_map *map);
 #ifdef CONFIG_MEMCG
 void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
 			 struct mem_cgroup **new_memcg);
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index c7bc9ba9b331f..abd0f67c68d97 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -602,15 +602,14 @@ static bool can_alloc_pages(void)
 		!IS_ENABLED(CONFIG_PREEMPT_RT);
 }
 
+#define BPF_PAGE_GFP (GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT | __GFP_NOWARN)
+
 static struct page *__bpf_alloc_page(int nid)
 {
 	if (!can_alloc_pages())
 		return alloc_pages_nolock(__GFP_ACCOUNT, nid, 0);
 
-	return alloc_pages_node(nid,
-				GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT
-				| __GFP_NOWARN,
-				0);
+	return alloc_pages_node(nid, BPF_PAGE_GFP, 0);
 }
 
 int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
@@ -636,6 +635,27 @@ int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
 	return ret;
 }
 
+/*
+ * Allocate a page for map memory with the blocking allocator so it can
+ * reclaim. __GFP_RETRY_MAYFAIL keeps it from invoking the OOM killer: the
+ * page is charged to the map's memcg, which need not be the caller's, so
+ * an OOM there could kill unrelated tasks in the map's cgroup while a
+ * foreign caller could never be its victim.
+ *
+ * bpf_map_alloc_pages() serves arbitrary BPF program context and stays
+ * reentrancy-safe by falling back to the non-blocking allocator. This
+ * helper always blocks, so a sleepable context alone is not enough: the
+ * caller must guarantee it is not already inside the page allocator or
+ * reclaim, where blocking here would reenter mm and deadlock. The only
+ * user is arena_vm_fault(), a userspace page fault in task context.
+ */
+struct page *bpf_map_alloc_page_sleepable(const struct bpf_map *map)
+{
+	might_sleep();
+	return alloc_pages_node(map->numa_node,
+				BPF_PAGE_GFP | __GFP_RETRY_MAYFAIL, 0);
+}
+
 static int btf_field_cmp(const void *a, const void *b)
 {
 	const struct btf_field *f1 = a, *f2 = b;
-- 
2.43.0