Re: [PATCH v2] oom: fix integer overflow of points in oom_badness

From: David Rientjes
Date: Mon Oct 31 2011 - 14:22:50 EST

On Mon, 31 Oct 2011, Frantisek Hrbata wrote:

> An integer overflow will happen on 64bit archs if task's sum of rss, swapents
> and nr_ptes exceeds (2^31)/1000 value. This was introduced by commit
> f755a04 oom: use pte pages in OOM score

This commit was introduced in 2.6.39 but also backported to stable since
2.6.36, so presumably we'd need to mark this for stable as well going back
that far.

> where the oom score computation was divided into several steps and it's no
> longer computed as one expression in unsigned long(rss, swapents, nr_pte are
> unsigned long), where the result value assigned to points(int) is in
> range(1..1000). So there could be an int overflow while computing
> 176 points *= 1000;
> and points may have negative value. Meaning the oom score for a mem hog task
> will be one.
> 196 if (points <= 0)
> 197 return 1;
> For example:
> [ 3366] 0 3366 35390480 24303939 5 0 0 oom01
> Out of memory: Kill process 3366 (oom01) score 1 or sacrifice child
> Here the oom1 process consumes more than 24303939(rss)*4096~=92GB physical
> memory, but it's oom score is one.
> In this situation the mem hog task is skipped and oom killer kills another and
> most probably innocent task with oom score greater than one.
> The points variable should be of type long instead of int to prevent the int
> overflow.
> Signed-off-by: Frantisek Hrbata <fhrbata@xxxxxxxxxx>

Acked-by: David Rientjes <rientjes@xxxxxxxxxx>
Cc: stable@xxxxxxxxxx [2.6.36+]
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at