[PATCH] mm: memcg: stop reclaim when a limit update is superseded

[Guopeng Zhang]

From: Guopeng Zhang <zhangguopeng@xxxxxxxxxx>

kernfs serializes file operations only per open file, so separate open
files can update the same memory.high or memory.max file concurrently.
Both handlers store the new limit before synchronous reclaim, but
continue to use the writer's local target in the reclaim loop. If another
writer raises or removes the limit, the first writer can continue
reclaiming toward a stale target.

For memory.max, this can leave the writer looping indefinitely once
reclaim retries are exhausted. The OOM path sees sufficient margin under
the current limit and returns true without killing, while the writer
still compares usage against its stale target and records another OOM
event.

Check the current limit at the start of each reclaim iteration and stop
if it no longer matches the writer's target.

Fixes: 8c8c383c04f6 ("mm: memcontrol: try harder to set a new memory.high")
Fixes: b6e6edcfa405 ("mm: memcontrol: reclaim and OOM kill when shrinking memory.max below usage")
Signed-off-by: Guopeng Zhang <zhangguopeng@xxxxxxxxxx>
---
Reproducer:

Populate a cgroup with anonymous memory and disable swapping. Lower
memory.max from one open file, then restore it to "max" through another
open file after the new limit becomes visible.

Without the patch, the first writer remains blocked and repeatedly
increments the OOM event counter. With the patch, it returns normally.

 mm/memcontrol.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 8319ad8c5c23..638bdc766616 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4798,6 +4798,9 @@ static ssize_t memory_high_write(struct kernfs_open_file *of,
 		unsigned long nr_pages = page_counter_read(&memcg->memory);
 		unsigned long reclaimed;
 
+		if (high != READ_ONCE(memcg->memory.high))
+			break;
+
 		if (nr_pages <= high)
 			break;
 
@@ -4853,6 +4856,9 @@ static ssize_t memory_max_write(struct kernfs_open_file *of,
 	for (;;) {
 		unsigned long nr_pages = page_counter_read(&memcg->memory);
 
+		if (max != READ_ONCE(memcg->memory.max))
+			break;
+
 		if (nr_pages <= max)
 			break;
 
-- 
2.25.1