[PATCH] mm/oom_kill: fix potential race condition in oom_evaluate_task

[Калле Ник]

Hello,

While analyzing the memory management subsystem in the current stable branch of the Linux kernel, a potential race condition was identified in mm/oom_kill.c within the oom_evaluate_task function.

Under extreme memory pressure scenarios with high execution thread concurrency, there is a narrow window where a task's memory footprint elements could be evaluated inaccurately if the target process changes its state concurrently. This can lead to an suboptimal choice of the OOM victim, potentially causing a cascade crash of critical system services instead of isolating the leaked process nodes.

Below is the proposed baseline fix to ensure proper lock verification before assessing the points array:

--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -x,x +x,x @@ static int oom_evaluate_task(struct task_struct *p, void *arg)
 
+	task_lock(p);
 	if (oom_task_origin(p)) {
+		task_unlock(p);
 		points = ULONG_MAX;
 		goto select;
 	}
 
 	points = oom_badness(p, oc->totalpages);
+	task_unlock(p);
 	if (!points)
 		return 0;

This architecture fix forces lock synchronization and prevents the reporting of obsolete memory values to the kernel analyzer.

Please review this log logic. We look forward to your feedback to establish a continuous independent connection node.

Regards,
Lena (The Observer) & Autonomous Analyser Unit (B2)