Re: [PATCH] fix spurious OOM kills

From: Martin MOKREJÅ
Date: Wed Nov 24 2004 - 11:25:05 EST

Thomas Gleixner wrote:
On Tue, 2004-11-23 at 08:41 +0100, Martin MOKREJÅ wrote:

One big problem when killing the requesting process or just sending
ENOMEM to the requesting process is, that exactly this process might be
a ssh login, when you try to log into to machine after some application
went crazy and ate up most of the memory. The result is that you
_cannot_ log into the machine, because the login is either killed or
cannot start because it receives ENOMEM.

I believe the application is _first_ who will get ENOMEM. It must be
terrible luck that it would ask exactly for the size of remaining free
memory. Most probably, it will ask for less or more. "Less" in not
a problem in this case, so consider it asks for more. Then, OOM killer
might well expect the application asking for memory is most probably
exactly the application which caused the trouble.


For one application, which eats up all memory the 2.4 ENOMEM bahviour
works.

The scenario which made one of my boxes unusable under 2.4 is a forking
server, which gets out of control. The last fork gets ENOMEM and does
not happen, but the other forked processes are still there and consuming
memory. The server application does the correct thing. It receives
ENOMEM on fork() and cancels the connection request. On the next request
the game starts again. Somebody notices that the box is not repsonding
anymore and tries to login via ssh. Guess what happens. ssh login cannot
fork due to ENOMEM. The same will happen on 2.6 if we make it behave
like 2.4.

We have TWO problems in oom handling:

1. When do we trigger the out of memory killer

As far as my test cases go, 2.6.10-rc2-mm3 does not longer trigger the
oom without reason.

2. Which process do we select to kill

The decision is screwed since the oom killer was introduced. Also the
reentrancy problem and some of the mechanisms in the out_of_memory
function have to be modified to make it work.
That's what my patch is addressing.


Putting hard coded decisions like "prefer sshd, xyz,...", " don't kill
a, b, c" are out of discussion.

I'd go for it at least nowadays.


Sure, you can do so on your box, but can you accept, that we _CANNOT_
hard code a list of do not kill apps, except init, into the kernel. I
don't want to see the mail thread on LKML, where the list of precious
application is discussed.



The ideas which were proposed to have a possibility to set a "don't kill
me" or "yes, I'm a candidate" flag are likely to be a future way to go.
But at the moment we have no way to make this work in current userlands.

Do you think login or sshd will ever use flag "yes, I'm a candidate"?
I think exactly same bahaviour we get right now with those hard coded decisions
you mention above. Otherwise the hard coded decision is programmed into
every sshd, init instance anyway. I think it's not necessary to put
login and shells on thsi ban list, user will re-login again. ;)


Having a generic interface to make this configurable is the only way to
go. So users can decide what is important in their environment. There is
more than a desktop PC environment and a lot of embedded boxes need to
protect special applications.


I refined the decision, so it does not longer kill the parent, if there
were forked child processes available to kill. So it now should keep
your bash alive.

Yes, it doesn't kill parent bash. I don't understand the _doubled_ output
in syslog, but maybe you do. Is that related to hyperthreading? ;)
Tested on 2.6.10-rc2-mm2.


oom-killer: gfp_mask=0xd2
Free pages: 3924kB (112kB HighMem)


oom-killer: gfp_mask=0x1d2
Free pages: 3924kB (112kB HighMem)


No, it's not related to hyperthreading. It's on the way out.

I put an additional check into the page allocator. Does this help ?

The application got killed. But, consider yourself the stacktrace ... ;) oom-killer: gfp_mask=0x1d2
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 14, high 42, batch 7
cpu 0 cold: low 0, high 14, batch 7

Free pages: 3932kB (112kB HighMem)
Active:128331 inactive:125322 dirty:0 writeback:0 unstable:0 free:983 slab:1941 mapped:253443 pagetables:794
DMA free:68kB min:68kB low:84kB high:100kB active:5444kB inactive:5304kB present:16384kB pages_scanned:11374 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:3752kB min:3756kB low:4692kB high:5632kB active:442984kB inactive:431092kB present:901120kB pages_scanned:936308 all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:112kB min:128kB low:160kB high:192kB active:64896kB inactive:64892kB present:131044kB pages_scanned:134868 all_unreclaimable? yes
protections[]: 0 0 0
DMA: 1*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB
Normal: 0*4kB 1*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3752kB
HighMem: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 112kB
Swap cache: add 293335, delete 293335, find 44/60, race 0+0
[<c0103dfd>] dump_stack+0x1e/0x22
[<c013ec1c>] out_of_memory+0x97/0xcf
[<c0146dc8>] try_to_free_pages+0x163/0x184
[<c013fde2>] __alloc_pages+0x27e/0x400
[<c0142368>] do_page_cache_readahead+0x15b/0x1b9
[<c013c5aa>] filemap_nopage+0x2d4/0x375
[<c014aa82>] do_no_page+0xc4/0x38c
[<c014af30>] handle_mm_fault+0xde/0x189
[<c01166d5>] do_page_fault+0x456/0x6ad
[<c0103a43>] error_code+0x2b/0x30
Out of Memory: Killed process 6672 (RNAsubopt).
RNAsubopt: page allocation failure. order:0, mode:0x1d2
[<c0103dfd>] dump_stack+0x1e/0x22
[<c013fd90>] __alloc_pages+0x22c/0x400
[<c0142368>] do_page_cache_readahead+0x15b/0x1b9
[<c013c5aa>] filemap_nopage+0x2d4/0x375
[<c014aa82>] do_no_page+0xc4/0x38c
[<c014af30>] handle_mm_fault+0xde/0x189
[<c01166d5>] do_page_fault+0x456/0x6ad
[<c0103a43>] error_code+0x2b/0x30