Re: System freezes after OOM

From: David Rientjes
Date: Thu Jul 14 2016 - 16:22:48 EST

On Thu, 14 Jul 2016, Mikulas Patocka wrote:

> > schedule
> > schedule_timeout
> > io_schedule_timeout
> > mempool_alloc
> > __split_and_process_bio
> > dm_request
> > generic_make_request
> > submit_bio
> > mpage_readpages
> > ext4_readpages
> > __do_page_cache_readahead
> > ra_submit
> > filemap_fault
> > handle_mm_fault
> > __do_page_fault
> > do_page_fault
> > page_fault
> Device mapper should be able to proceed if there is no available memory.
> If it doesn't proceed, there is a bug in it.

The above stack trace has nothing to do with the device mapper with
pre-f9054c70d28b behavior. It simply is calling into mempool_alloc() and
no elements are being returned to the mempool that allow it to return.

Recall that in the above situation, the whole system is oom; nothing can
allocate memory. The oom killer has selected the above process to be oom
killed, so all other processes on the system trying to allocate memory
will stall in the page allocator waiting for this process to exit.

The natural response to this situation is to allow access to memory
reserves, if possible, so that mempool_alloc() may return. There is no
guarantee that _anything_ can return memory to the mempool, especially in
a system oom condition where nothing can make forward progress. Insisting
that should be guaranteed is not helpful.

> I'd like to ask - what device mapper targets did you use in this case? Are
> there some other deadlocked processes? (show sysrq-t, sysrq-w when this
> happened)

Every process on the system is deadlocked because they cannot get memory
through the page allocator until the above process exits. That is how the
oom killer works: select a process, kill it, give it access to memory
reserves so it may exit and free its memory, and wait.

> Did the machine lock up completely with that stacktrace, or was it just
> slowed down?

Hundreds of machines locked up and rebooted after a two hour watchdog

> > That's obviously not the case if we have hundreds of machines timing out
> > after two hours waiting for that fault to succeed. The mempool interface
> > cannot require that users return elements to the pool synchronous with all
> > allocators so that we can happily loop forever, the only requirement on
> Mempool users must return objects to the mempool.

Not possible when the system is livelocked.