Re: [RFC PATCH] mm/oom_kill: allow oom kill allocating task for non-global case
From: Waiman Long
Date: Mon Jun 07 2021 - 16:42:47 EST
On 6/7/21 3:36 PM, Michal Hocko wrote:
On Mon 07-06-21 15:18:38, Waiman Long wrote:
On 6/7/21 3:04 PM, Michal Hocko wrote:
On Mon 07-06-21 14:51:05, Waiman Long wrote:
On 6/7/21 2:43 PM, Shakeel Butt wrote:
On Mon, Jun 7, 2021 at 9:45 AM Waiman Long <llong@xxxxxxxxxx> wrote:
On 6/7/21 12:31 PM, Aaron Tomlin wrote:
At the present time, in the context of memcg OOM, even when
sysctl_oom_kill_allocating_task is enabled/or set, the "allocating"
task cannot be selected, as a target for the OOM killer.
This patch removes the restriction entirely.
Signed-off-by: Aaron Tomlin <atomlin@xxxxxxxxxx>
---
mm/oom_kill.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index eefd3f5fde46..3bae33e2d9c2 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *oc)
oc->nodemask = NULL;
check_panic_on_oom(oc);
- if (!is_memcg_oom(oc) && sysctl_oom_kill_allocating_task &&
- current->mm && !oom_unkillable_task(current) &&
- oom_cpuset_eligible(current, oc) &&
+ if (sysctl_oom_kill_allocating_task && current->mm &&
+ !oom_unkillable_task(current) &&
+ oom_cpuset_eligible(current, oc) &&
current->signal->oom_score_adj != OOM_SCORE_ADJ_MIN) {
get_task_struct(current);
oc->chosen = current;
To provide more context for this patch, we are actually seeing that in a
customer report about OOM happened in a container where the dominating
task used up most of the memory and it happened to be the task that
triggered the OOM with the result that no killable process could be
found.
Why was there no killable process? What about the process allocating
the memory or is this remote memcg charging?
It is because the other processes have a oom_adjust_score of -1000. So they
are non-killable. Anyway, they don't consume that much memory and killing
them won't free up that much.
The other process that uses most of the memory is the one that trigger the
OOM kill in the first place because the memory limit has been reached in new
memory allocation. Based on the current logic, this process cannot be killed
at all even if we set the oom_kill_allocating_task to 1 if the OOM happens
only within the memcg context, not in a global OOM situation. This patch is
to allow this process to be killed under this circumstance.
Do you have the oom report? I do not see why the allocating task hasn't
been chosen.
A partial OOM report below:
Do you happen to have the full report?
I need to ask to see if I can release the full report.
[ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116
:
[ 8227.239769] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 8227.242495] [1611298] 0 1611298 35869 635 167936 0 -1000 conmon
[ 8227.242518] [1702509] 0 1702509 35869 701 176128 0 -1000 conmon
[ 8227.242522] [1703345] 1001050000 1703294 183440 0 2125824 0 999 node
[ 8227.242706] Out of memory and no killable processes...
[ 8227.242731] node invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=999
[ 8227.242732] node cpuset=crio-b8ac7e23f7b520c0365461defb66738231918243586e287bfb9e206bb3a0227a.scope mems_allowed=0-1
So in this case, node cannot kill itself and no other processes are
available to be killed.
The process is clearly listed as eligible so the oom killer should find
it and if it hasn't then this should be investigated. Which kernel is
this?
Right. I don't know why the current cannot be selected. I think we may
need to enhance the OOM but no killable process report to list the
reason a task is skipped other than oom_score_adj. The kernel is a
RHEL8.2 kernel which has OOM code pretty close to upstream.
Cheers,
Longman