[PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Ye Liu]

From: Ye Liu <liuye@xxxxxxxxxx>

The vmap_purge_lock mutex can be held for an extended period by
__purge_vmap_area_lazy() which calls flush_work() to wait for
purge_vmap_node workers while holding the lock.  Under memory
pressure, those workers may themselves be blocked in direct
reclaim trying to acquire the same lock via the
vmap_node_shrink_scan() shrinker callback, creating a circular
dependency that deadlocks the entire system.

Two places acquire vmap_purge_lock from paths that can be reached
during direct reclaim:

1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
   mutex_trylock().  This is a shrinker that only decays the vmap
   pool and returns SHRINK_STOP without freeing memory; skipping a
   decay cycle when the lock is contended is harmless and prevents
   tasks from piling up on the mutex in the direct reclaim path.

2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
   mutex_trylock().  This is called from the vmalloc allocation
   overflow path; if trylock fails, another thread is already
   purging and the allocator's retry will find freed space.  The
   notifier chain provides a fallback if the retry still fails.

Both trylock failures break the circular dependency: the lock
holder's flush_work() can complete because workers are no longer
blocked on vmap_purge_lock in the direct reclaim path.

Fixes: 7679ba6b36db ("mm: vmalloc: add a shrinker to drain vmap pools")
Suggested-by: Uladzislau Rezki <urezki@xxxxxxxxx>
Suggested-by: Dev Jain <dev.jain@xxxxxxx>
Signed-off-by: Ye Liu <liuye@xxxxxxxxxx>
---
v2:
 - Use mutex_trylock instead of mutex_lock to acquire vmap_purge_lock,
   as suggested by Uladzislau Rezki and Dev Jain.
 - Link: https://lore.kernel.org/all/20260824095020.1225189-1-ye.liu@xxxxxxxxx/
 mm/vmalloc.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index bea9f76ed7e7..e5c68b795a3e 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -2440,7 +2440,8 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
 static void reclaim_and_purge_vmap_areas(void)
 
 {
-	mutex_lock(&vmap_purge_lock);
+	if (!mutex_trylock(&vmap_purge_lock))
+		return;
 	purge_fragmented_blocks_allcpus();
 	__purge_vmap_area_lazy(ULONG_MAX, 0, true);
 	mutex_unlock(&vmap_purge_lock);
@@ -5519,10 +5520,20 @@ vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	struct vmap_node *vn;
 
-	guard(mutex)(&vmap_purge_lock);
+	/*
+	 * This shrinker is invoked from direct reclaim where memory
+	 * pressure is already high.  Blocking on vmap_purge_lock here
+	 * can deadlock the system: the lock holder may be blocked in
+	 * flush_work() waiting for a worker that is stuck in this same
+	 * reclaim path trying to acquire the same lock.  Use trylock
+	 * to avoid this; skipping a pool decay cycle is harmless.
+	 */
+	if (!mutex_trylock(&vmap_purge_lock))
+		return SHRINK_STOP;
 	for_each_vmap_node(vn)
 		decay_va_pool_node(vn, true);
 
+	mutex_unlock(&vmap_purge_lock);
 	return SHRINK_STOP;
 }
 
-- 
2.25.1