[PATCH 2/2] mm/mempolicy: skip task mempolicy for unmovable unaccounted kernel allocations

[Gregory Price]

Bare calls to alloc_pages / folio_alloc / vmalloc / slab-refill cause
vma-less allocations which fall back to the current task's mempolicy
for a placement hint.  This has the effect of directing incidental,
global, task-unrelated allocations to follow the task's mempolicy.

On uniform multi-socket systems, this is not implicitly harmful.
On tiered memory systems, these incidental allocations can land
on slower memory tiers and actively cause performance regressions.

For example, the code in lib/stackdepot.c allocates a global
resource that sticks around for the life of the system:

  stack_depot_save_flags()
      page = alloc_pages(gfp_nested_mask(alloc_flags));

In this example:
  1) the ownership is global (stackdepot, no association to the task)
  2) The lifetime is forever (the pool is more or less permanent)
  3) Whichever task causes the FIRST allocation to occur dictates
     placement for future users - permanently.
  4) It's unaccounted and unmovable.

While this doesn't prevent fallback allocations occurring and landing
in the same class of memory the same way, a per-task interleave policy
shouldn't dictate placement for such allocations.

Make a new helper, alloc_task_policy(), to identify three classes of
allocations we explicitly do not want to follow the task policy.

 1) in_interrupt() - interrupt allocations are not related to the task
 2) __GFP_THISNODE - caller explicit placement
 3) unmovable, unaccounted kernel allocations
    !(__GFP_MOVABLE | __GFP_ACCOUNT)

The third filter keeps incidental global kernel allocations from being
spread throughout the system when a task has set a mempolicy. The other
two are existing filters consolidated by the helper.

Allowing movable allocations to follow task policy retains existing
non-harmful placement behavior, and cpuset constraints are still
enforced by ALLOC_CPUSET.  Pagecache placement is made explicit by
the preceding patch.

Everything else (unmovable and unaccounted) falls back to node-local.

User-space folios are unaffected: they fall through folio_alloc_mpol().

Suggested-by: Johannes Weiner <hannes@xxxxxxxxxxx>
Link: https://lore.kernel.org/linux-mm/akU2_Mt_JDCxmnc7@xxxxxxxxxxx/
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Gregory Price <gourry@xxxxxxxxxx>
---
 mm/mempolicy.c | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index fc9051e9558b..6c00dc32c8b4 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2737,16 +2737,35 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct
 }
 EXPORT_SYMBOL(vma_alloc_folio_noprof);
 
-struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order)
+/*
+ * Bare alloc_pages() calls only have task policy as a placement hint.
+ * There are 3 scenarios where we don't want to use the task policy:
+ *  1) interrupt context - these allocations are not related to the task
+ *  2) __GFP_THISNODE - the caller asked for explicit placement
+ *  3) Unmovable (!__GFP_MOVABLE), unaccounted (!__GFP_ACCOUNT) kernel memory
+ *
+ * The third filter keeps incidental global kernel allocations from being
+ * spread throughout the system when a task has set an interleave policy.
+ * This can include cached global metadata for drivers and internal
+ * services that outlives the life of the task.
+ *
+ * Allowing movable allocations to follow task policy retains existing
+ * non-harmful behavior, and cpuset constraints are still respected.
+ *
+ * No reference counting needed for current->mempolicy nor default_policy
+ */
+static struct mempolicy *alloc_task_policy(gfp_t gfp)
 {
-	struct mempolicy *pol = &default_policy;
+	if (in_interrupt() || (gfp & __GFP_THISNODE))
+		return &default_policy;
+	if (!(gfp & (__GFP_MOVABLE | __GFP_ACCOUNT)))
+		return &default_policy;
+	return get_task_policy(current);
+}
 
-	/*
-	 * No reference counting needed for current->mempolicy
-	 * nor system default_policy
-	 */
-	if (!in_interrupt() && !(gfp & __GFP_THISNODE))
-		pol = get_task_policy(current);
+struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order)
+{
+	struct mempolicy *pol = alloc_task_policy(gfp);
 
 	return alloc_pages_mpol(gfp, order, pol, NO_INTERLEAVE_INDEX,
 				       numa_node_id());
@@ -2964,13 +2983,10 @@ static unsigned long alloc_pages_bulk_preferred_many(gfp_t gfp, int nid,
 unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,
 		unsigned long nr_pages, struct page **page_array)
 {
-	struct mempolicy *pol = &default_policy;
+	struct mempolicy *pol = alloc_task_policy(gfp);
 	nodemask_t *nodemask;
 	int nid;
 
-	if (!in_interrupt() && !(gfp & __GFP_THISNODE))
-		pol = get_task_policy(current);
-
 	if (pol->mode == MPOL_INTERLEAVE)
 		return alloc_pages_bulk_interleave(gfp, pol,
 							 nr_pages, page_array);
-- 
2.53.0-Meta