On 12/09/2016 07:01 PM, Gerhard Wiesinger wrote:
On 09.12.2016 18:30, Michal Hocko wrote:We don't have such bulletproof reserves. In this case the amount of
On Fri 09-12-16 17:58:14, Gerhard Wiesinger wrote:I'm not familiar with the Linux implementation of the VM system in
On 09.12.2016 17:09, Michal Hocko wrote:[...]
Well this is what we try and call it memory reclaim. But if we are notYes, it might be large on the update situation, but that should be handled[97883.882611] Mem-Info:there is still some page cache which doesn't seem to be neither dirty
[97883.883747] active_anon:2915 inactive_anon:3376 isolated_anon:0
active_file:3902 inactive_file:3639 isolated_file:0
unevictable:0 dirty:205 writeback:0 unstable:0
slab_reclaimable:9856 slab_unreclaimable:9682
mapped:3722 shmem:59 pagetables:2080 bounce:0
free:748 free_pcp:15 free_cma:0
nor under writeback. So it should be theoretically reclaimable but for
some reason we cannot seem to reclaim that memory.
There is still some anonymous memory and free swap so we could reclaim
it as well but it all seems pretty down and the memory pressure is
really large
by a virtual memory system by the kernel, right?
able to reclaim anything then we eventually have to give up and trigger
the OOM killer.
detail. But can't you reserve as much memory for the kernel (non
pageable) at least that you can swap everything out (even without
killing a process at least as long there is enough swap available, which
should be in all of my cases)?
anonymous memory that can be swapped out is relatively low, and either
something is pinning it in memory, or it's being swapped back in quickly.
Well it's not really a surprise that if the VM is small enough andNow the information that 4.4 made a difference isI'm using ext4 only with virt-* drivers (storage, network). But it is
interesting. I do not really see any major differences in the reclaim
between 4.3 and 4.4 kernels. The reason might be somewhere else as well.
E.g. some of the subsystem consumes much more memory than before.
Just curious, what kind of filesystem are you using?
definitly a virtual memory allocation/swap usage issue.
Could you try someActivated it. But I think it should be very easy to trigger also on your
additional debugging. Enabling reclaim related tracepoints might tell us
more. The following should tell us more
mount -t tracefs none /trace
echo 1 > /trace/events/vmscan/enable
echo 1 > /trace/events/writeback/writeback_congestion_wait/enable
cat /trace/trace_pipe > trace.log
Collecting /proc/vmstat over time might be helpful as well
mkdir logs
while true
do
cp /proc/vmstat vmstat.$(date +%s)
sleep 1s
done
side. A very small configured VM with a program running RAM
allocations/writes (I guess you have some testing programs already)
should be sufficient to trigger it. You can also use the attached
program which I used to trigger such situations some years ago. If it
doesn't help try to reduce the available CPU for the VM and also I/O
(e.g. use all CPU/IO on the host or other VMs).
workload large enough, OOM killer will kick in. The exact threshold
might have changed between kernel versions for a number of possible reasons.
BTW: Don't know if you have seen also my original message on the kernelYeah we were involved in the last one. The regressions were about
mailinglist only:
Linus had also OOM problems with 1kB RAM requests and a lot of free RAM
(use a translation service for the german page):
https://lkml.org/lkml/2016/11/30/64
https://marius.bloggt-in-braunschweig.de/2016/11/17/linuxkernel-4-74-8-und-der-oom-killer/
https://www.spinics.net/lists/linux-mm/msg113661.html
high-order allocations
though (the 1kB premise turned out to be misinterpretation) and there
were regressions
for those in 4.7/4.8. But yours are order-0.