OOM: Better, but still there on 4.9
From: Nils Holland
Date: Thu Dec 15 2016 - 18:06:49 EST
Hi folks,
I've been reading quite a bit about OOM related issues in recent
kernels, and as I've been experiencing some of these myself for quite
a while, I thought I'd send in my report in case in the hope that it
is useful. Of course, if there's ever anything to test, like some
patches or something, I'd be glad to help!
Now, my situation: I have two different x86 machines, both equipped
with 4 GB of RAM, running 32 bit kernels. I've never really observed
any OOM issues until kernel 4.8, but with that kernel, it was enough
to unpack a bigger source tarball (like the firefox sources) on a
freshly booted system and subsequently compile them, and with very
high certainty, the OOL killer would kick in during the compile,
killing a whole lot of processes, with the machine then becoming
unresponsive and finally a kernel panic taking place.
With kernel 4.9, these OOM events seem to be somewhat harder to
trigger, but in most cases, unpacking some larger tarballs and then
launching a build process on a freshly booted system without many
other processed (not even X) running seems to do the trick. However,
the consequences don't seem to be as severe as they were in 4.8: The
machines did, in fact, become unresponsive in the way that logging in
locally (after I'm being thrown out when my bash gets killed by the
OOM reaper) is no longer possible, sshing into the machine also
doesn't work anymore (most likely because sshd has also been killed),
but in all cases the machine was still pingable and the magic
SysRequest key combo was still working. I've not yet seen a single
real kernel panic as I did with 4.8, still, the only way to get the
machine back into action was a hard reboot via the magic SysRequest
commands or a power cycle.
For the reference, I'm attaching an OOM I've observed under 4.9 at the
end of this machine. This one happened after I had been using the
machine in a normal fashion for a short time, and then had portage,
Gentoo's build system, unpack the firefox sources - compiling hadn't
even started yet at this point. Oh yes, I'm using btrfs in case that
might make a differences - at least I've seen some references to it in
other similar reports I've found on the web.
Of course, none of this are workloads that are new / special in any
way - prior to 4.8, I never experienced any issues doing the exact
same things.
Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, oom_score_adj=0
Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0
Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted 4.9.0-gentoo #2
Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014
Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1)
Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 00000000 eff0b634 c1163332 00000000 00000292
Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 e7fa2900 c1b58785 eff0b734
Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 00000007 00000000 00000000
Dec 15 19:02:18 teela kernel: Call Trace:
Dec 15 19:02:18 teela kernel: [<c142bcce>] dump_stack+0x47/0x69
Dec 15 19:02:18 teela kernel: [<c1163332>] dump_header+0x60/0x178
Dec 15 19:02:18 teela kernel: [<c1431876>] ? ___ratelimit+0x86/0xe0
Dec 15 19:02:18 teela kernel: [<c110795f>] oom_kill_process+0x20f/0x3d0
Dec 15 19:02:18 teela kernel: [<c1043895>] ? has_capability_noaudit+0x15/0x20
Dec 15 19:02:18 teela kernel: [<c11075c7>] ? oom_badness.part.13+0xb7/0x130
Dec 15 19:02:18 teela kernel: [<c1107df9>] out_of_memory+0xd9/0x260
Dec 15 19:02:18 teela kernel: [<c110ba0b>] __alloc_pages_nodemask+0xbfb/0xc80
Dec 15 19:02:18 teela kernel: [<c110414d>] pagecache_get_page+0xad/0x270
Dec 15 19:02:18 teela kernel: [<c13664a6>] alloc_extent_buffer+0x116/0x3e0
Dec 15 19:02:18 teela kernel: [<c1334a2e>] btrfs_find_create_tree_block+0xe/0x10
Dec 15 19:02:18 teela kernel: [<c132a57f>] btrfs_alloc_tree_block+0x1ef/0x5f0
Dec 15 19:02:18 teela kernel: [<c130f7c3>] __btrfs_cow_block+0x143/0x5f0
Dec 15 19:02:18 teela kernel: [<c130fe1a>] btrfs_cow_block+0x13a/0x220
Dec 15 19:02:18 teela kernel: [<c13132f1>] btrfs_search_slot+0x1d1/0x870
Dec 15 19:02:18 teela kernel: [<c132fcdd>] btrfs_lookup_file_extent+0x4d/0x60
Dec 15 19:02:18 teela kernel: [<c1354fe6>] __btrfs_drop_extents+0x176/0x1070
Dec 15 19:02:18 teela kernel: [<c1150377>] ? kmem_cache_alloc+0xb7/0x190
Dec 15 19:02:18 teela kernel: [<c133dbb5>] ? start_transaction+0x65/0x4b0
Dec 15 19:02:18 teela kernel: [<c1150597>] ? __kmalloc+0x147/0x1e0
Dec 15 19:02:18 teela kernel: [<c1345005>] cow_file_range_inline+0x215/0x6b0
Dec 15 19:02:18 teela kernel: [<c13459fc>] cow_file_range.isra.49+0x55c/0x6d0
Dec 15 19:02:18 teela kernel: [<c1361795>] ? lock_extent_bits+0x75/0x1e0
Dec 15 19:02:18 teela kernel: [<c1346d51>] run_delalloc_range+0x441/0x470
Dec 15 19:02:18 teela kernel: [<c13626e4>] writepage_delalloc.isra.47+0x144/0x1e0
Dec 15 19:02:18 teela kernel: [<c1364548>] __extent_writepage+0xd8/0x2b0
Dec 15 19:02:18 teela kernel: [<c1365c4c>] extent_writepages+0x25c/0x380
Dec 15 19:02:18 teela kernel: [<c1342cd0>] ? btrfs_real_readdir+0x610/0x610
Dec 15 19:02:18 teela kernel: [<c133ff0f>] btrfs_writepages+0x1f/0x30
Dec 15 19:02:18 teela kernel: [<c110ff85>] do_writepages+0x15/0x40
Dec 15 19:02:18 teela kernel: [<c1190a95>] __writeback_single_inode+0x35/0x2f0
Dec 15 19:02:18 teela kernel: [<c119112e>] writeback_sb_inodes+0x16e/0x340
Dec 15 19:02:18 teela kernel: [<c119145a>] wb_writeback+0xaa/0x280
Dec 15 19:02:18 teela kernel: [<c1191de8>] wb_workfn+0xd8/0x3e0
Dec 15 19:02:18 teela kernel: [<c104fd34>] process_one_work+0x114/0x3e0
Dec 15 19:02:18 teela kernel: [<c1050b4f>] worker_thread+0x2f/0x4b0
Dec 15 19:02:18 teela kernel: [<c1050b20>] ? create_worker+0x180/0x180
Dec 15 19:02:18 teela kernel: [<c10552e7>] kthread+0x97/0xb0
Dec 15 19:02:18 teela kernel: [<c1055250>] ? __kthread_parkme+0x60/0x60
Dec 15 19:02:18 teela kernel: [<c19b5cb7>] ret_from_fork+0x1b/0x28
Dec 15 19:02:18 teela kernel: Mem-Info:
Dec 15 19:02:18 teela kernel: active_anon:58685 inactive_anon:90 isolated_anon:0
active_file:274324 inactive_file:281962 isolated_file:0
unevictable:0 dirty:649 writeback:0 unstable:0
slab_reclaimable:40662 slab_unreclaimable:17754
mapped:7382 shmem:202 pagetables:351 bounce:0
free:206736 free_pcp:332 free_cma:0
Dec 15 19:02:18 teela kernel: Node 0 active_anon:234740kB inactive_anon:360kB active_file:1097296kB inactive_file:1127848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:29528kB dirty:2596kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 184320kB anon_thp: 808kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Dec 15 19:02:18 teela kernel: DMA free:3952kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:7316kB inactive_file:0kB unevictable:0kB writepending:96kB present:15992kB managed:15916kB mlocked:0kB slab_reclaimable:3200kB slab_unreclaimable:1408kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 813 3474 3474
Dec 15 19:02:18 teela kernel: Normal free:41332kB min:41368kB low:51708kB high:62048kB active_anon:0kB inactive_anon:0kB active_file:532748kB inactive_file:44kB unevictable:0kB writepending:24kB present:897016kB managed:836248kB mlocked:0kB slab_reclaimable:159448kB slab_unreclaimable:69608kB kernel_stack:1112kB pagetables:1404kB bounce:0kB free_pcp:528kB local_pcp:340kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 0 21292 21292
Dec 15 19:02:18 teela kernel: HighMem free:781660kB min:512kB low:34356kB high:68200kB active_anon:234740kB inactive_anon:360kB active_file:557232kB inactive_file:1127804kB unevictable:0kB writepending:2592kB present:2725384kB managed:2725384kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:800kB local_pcp:608kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 0 0 0
Dec 15 19:02:18 teela kernel: DMA: 0*4kB 2*8kB (ME) 5*16kB (UME) 13*32kB (UM) 11*64kB (UME) 3*128kB (UM) 1*256kB (M) 2*512kB (E) 1*1024kB (M) 0*2048kB 0*4096kB = 3904kB
Dec 15 19:02:18 teela kernel: Normal: 27*4kB (ME) 25*8kB (UME) 442*16kB (UME) 189*32kB (UME) 411*64kB (UME) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41396kB
Dec 15 19:02:18 teela kernel: HighMem: 1*4kB (M) 11*8kB (U) 2*16kB (U) 3*32kB (UM) 16*64kB (U) 1*128kB (M) 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 190*4096kB (UM) = 781660kB
Dec 15 19:02:18 teela kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 15 19:02:18 teela kernel: 556515 total pagecache pages
Dec 15 19:02:18 teela kernel: 0 pages in swap cache
Dec 15 19:02:18 teela kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 15 19:02:18 teela kernel: Free swap = 8191996kB
Dec 15 19:02:18 teela kernel: Total swap = 8191996kB
Dec 15 19:02:18 teela kernel: 909598 pages RAM
Dec 15 19:02:18 teela kernel: 681346 pages HighMem/MovableOnly
Dec 15 19:02:18 teela kernel: 15211 pages reserved
Dec 15 19:02:18 teela kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 15 19:02:18 teela kernel: [ 1888] 0 1888 6161 1112 10 3 0 0 systemd-journal
Dec 15 19:02:18 teela kernel: [ 2508] 0 2508 2959 945 6 3 0 -1000 systemd-udevd
Dec 15 19:02:18 teela kernel: [ 2610] 105 2610 3870 899 8 3 0 0 systemd-timesyn
Dec 15 19:02:18 teela kernel: [ 2613] 0 2613 6300 948 10 3 0 0 rsyslogd
Dec 15 19:02:18 teela kernel: [ 2615] 88 2615 1158 568 6 3 0 0 nullmailer-send
Dec 15 19:02:18 teela kernel: [ 2618] 0 2618 1514 1027 7 3 0 0 systemd-logind
Dec 15 19:02:18 teela kernel: [ 2619] 101 2619 1266 847 6 3 0 -900 dbus-daemon
Dec 15 19:02:18 teela kernel: [ 2620] 0 2620 622 300 5 3 0 0 atd
Dec 15 19:02:18 teela kernel: [ 2628] 0 2628 26097 3193 27 3 0 0 NetworkManager
Dec 15 19:02:18 teela kernel: [ 2647] 0 2647 1511 458 5 3 0 0 fcron
Dec 15 19:02:18 teela kernel: [ 2673] 0 2673 750 543 6 3 0 0 dhcpcd
Dec 15 19:02:18 teela kernel: [ 2676] 0 2676 638 447 5 3 0 0 vnstatd
Dec 15 19:02:18 teela kernel: [ 2690] 0 2690 1457 1061 6 3 0 -1000 sshd
Dec 15 19:02:18 teela kernel: [ 2716] 106 2716 16384 4239 20 3 0 0 polkitd
Dec 15 19:02:18 teela kernel: [ 2717] 0 2717 2145 1360 7 3 0 0 wpa_supplicant
Dec 15 19:02:18 teela kernel: [ 2947] 0 2947 1794 775 7 3 0 0 screen
Dec 15 19:02:18 teela kernel: [ 2950] 0 2950 1831 913 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: [ 2954] 0 2954 37411 36100 79 3 0 0 emerge
Dec 15 19:02:18 teela kernel: [ 2970] 0 2970 1152 507 6 3 0 0 agetty
Dec 15 19:02:18 teela kernel: [ 3897] 250 3897 548 358 5 3 0 0 sandbox
Dec 15 19:02:18 teela kernel: [ 3906] 250 3906 2625 1584 8 3 0 0 ebuild.sh
Dec 15 19:02:18 teela kernel: [ 3926] 250 3926 2657 1370 7 3 0 0 ebuild.sh
Dec 15 19:02:18 teela kernel: [ 3935] 250 3935 17160 16891 37 3 0 0 xz
Dec 15 19:02:18 teela kernel: [ 3936] 250 3936 799 510 5 3 0 0 tar
Dec 15 19:02:18 teela kernel: [ 4117] 0 4117 2598 1389 9 3 0 0 sshd
Dec 15 19:02:18 teela kernel: [ 4119] 0 4119 1964 1243 7 3 0 0 systemd
Dec 15 19:02:18 teela kernel: [ 4144] 0 4144 6645 632 10 3 0 0 (sd-pam)
Dec 15 19:02:18 teela kernel: [ 4163] 0 4163 1830 909 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: [ 4182] 0 4182 1695 684 7 3 0 0 screen
Dec 15 19:02:18 teela kernel: [ 4221] 0 4221 1831 893 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: Out of memory: Kill process 2954 (emerge) score 11 or sacrifice child
Dec 15 19:02:18 teela kernel: Killed process 3897 (sandbox) total-vm:2192kB, anon-rss:128kB, file-rss:1304kB, shmem-rss:0kB
Dec 15 19:02:18 teela kernel: bash invoked oom-killer: gfp_mask=0x27080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=0, order=1, oom_score_adj=0
Dec 15 19:02:18 teela kernel: bash cpuset=/ mems_allowed=0
Dec 15 19:02:18 teela kernel: CPU: 0 PID: 4221 Comm: bash Not tainted 4.9.0-gentoo #2
Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014
Dec 15 19:02:18 teela kernel: c5187d68 c142bcce c5187e98 00000000 c5187d98 c1163332 00000000 00200282
Dec 15 19:02:18 teela kernel: c5187d98 c1431876 c5187d9c e7fb0b00 e7fa2900 e7fa2900 c1b58785 c5187e98
Dec 15 19:02:18 teela kernel: c5187ddc c110795f c1043895 c5187dc8 c11075c7 00000007 00000000 00000000
Dec 15 19:02:18 teela kernel: Call Trace:
Dec 15 19:02:18 teela kernel: [<c142bcce>] dump_stack+0x47/0x69
Dec 15 19:02:18 teela kernel: [<c1163332>] dump_header+0x60/0x178
Dec 15 19:02:18 teela kernel: [<c1431876>] ? ___ratelimit+0x86/0xe0
Dec 15 19:02:18 teela kernel: [<c110795f>] oom_kill_process+0x20f/0x3d0
Dec 15 19:02:18 teela kernel: [<c1043895>] ? has_capability_noaudit+0x15/0x20
Dec 15 19:02:18 teela kernel: [<c11075c7>] ? oom_badness.part.13+0xb7/0x130
Dec 15 19:02:18 teela kernel: [<c1107df9>] out_of_memory+0xd9/0x260
Dec 15 19:02:18 teela kernel: [<c110b989>] __alloc_pages_nodemask+0xb79/0xc80
Dec 15 19:02:18 teela kernel: [<c1038a05>] copy_process.part.51+0xe5/0x1420
Dec 15 19:02:18 teela kernel: [<c1150377>] ? kmem_cache_alloc+0xb7/0x190
Dec 15 19:02:18 teela kernel: [<c1039ee7>] _do_fork+0xc7/0x360
Dec 15 19:02:18 teela kernel: [<c1182a4b>] ? fd_install+0x1b/0x20
Dec 15 19:02:18 teela kernel: [<c103a247>] SyS_clone+0x27/0x30
Dec 15 19:02:18 teela kernel: [<c10018bc>] do_fast_syscall_32+0x7c/0x130
Dec 15 19:02:18 teela kernel: [<c19b5d2b>] sysenter_past_esp+0x40/0x6a
Dec 15 19:02:18 teela kernel: Mem-Info:
Dec 15 19:02:18 teela kernel: active_anon:57050 inactive_anon:90 isolated_anon:0
active_file:274371 inactive_file:281954 isolated_file:0
unevictable:0 dirty:616 writeback:0 unstable:0
slab_reclaimable:40669 slab_unreclaimable:17758
mapped:7370 shmem:202 pagetables:346 bounce:0
free:208199 free_pcp:501 free_cma:0
Dec 15 19:02:18 teela kernel: Node 0 active_anon:228200kB inactive_anon:360kB active_file:1097484kB inactive_file:1127816kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:29480kB dirty:2464kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 184320kB anon_thp: 808kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Dec 15 19:02:18 teela kernel: DMA free:3904kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:7356kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15916kB mlocked:0kB slab_reclaimable:3208kB slab_unreclaimable:1448kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 813 3474 3474
Dec 15 19:02:18 teela kernel: Normal free:41280kB min:41368kB low:51708kB high:62048kB active_anon:0kB inactive_anon:0kB active_file:532796kB inactive_file:44kB unevictable:0kB writepending:144kB present:897016kB managed:836248kB mlocked:0kB slab_reclaimable:159468kB slab_unreclaimable:69584kB kernel_stack:1104kB pagetables:1384kB bounce:0kB free_pcp:628kB local_pcp:188kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 0 21292 21292
Dec 15 19:02:18 teela kernel: HighMem free:787612kB min:512kB low:34356kB high:68200kB active_anon:228200kB inactive_anon:360kB active_file:557332kB inactive_file:1127772kB unevictable:0kB writepending:2320kB present:2725384kB managed:2725384kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:1376kB local_pcp:648kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 0 0 0
Dec 15 19:02:18 teela kernel: DMA: 0*4kB 2*8kB (ME) 5*16kB (UME) 13*32kB (UM) 11*64kB (UME) 3*128kB (UM) 1*256kB (M) 2*512kB (E) 1*1024kB (M) 0*2048kB 0*4096kB = 3904kB
Dec 15 19:02:18 teela kernel: Normal: 26*4kB (M) 25*8kB (UM) 441*16kB (UM) 188*32kB (UM) 410*64kB (UME) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41280kB
Dec 15 19:02:18 teela kernel: HighMem: 37*4kB (UM) 21*8kB (UM) 6*16kB (UM) 8*32kB (UM) 22*64kB (UM) 5*128kB (M) 4*256kB (M) 3*512kB (M) 2*1024kB (M) 1*2048kB (M) 190*4096kB (UM) = 787612kB
Dec 15 19:02:18 teela kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 15 19:02:18 teela kernel: 556527 total pagecache pages
Dec 15 19:02:18 teela kernel: 0 pages in swap cache
Dec 15 19:02:18 teela kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 15 19:02:18 teela kernel: Free swap = 8191996kB
Dec 15 19:02:18 teela kernel: Total swap = 8191996kB
Dec 15 19:02:18 teela kernel: 909598 pages RAM
Dec 15 19:02:18 teela kernel: 681346 pages HighMem/MovableOnly
Dec 15 19:02:18 teela kernel: 15211 pages reserved
Dec 15 19:02:18 teela kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 15 19:02:18 teela kernel: [ 1888] 0 1888 6161 1112 10 3 0 0 systemd-journal
Dec 15 19:02:18 teela kernel: [ 2508] 0 2508 2959 945 6 3 0 -1000 systemd-udevd
Dec 15 19:02:18 teela kernel: [ 2610] 105 2610 3870 899 8 3 0 0 systemd-timesyn
Dec 15 19:02:18 teela kernel: [ 2613] 0 2613 6300 951 10 3 0 0 rsyslogd
Dec 15 19:02:18 teela kernel: [ 2615] 88 2615 1158 568 6 3 0 0 nullmailer-send
Dec 15 19:02:18 teela kernel: [ 2618] 0 2618 1514 1027 7 3 0 0 systemd-logind
Dec 15 19:02:18 teela kernel: [ 2619] 101 2619 1266 847 6 3 0 -900 dbus-daemon
Dec 15 19:02:18 teela kernel: [ 2620] 0 2620 622 300 5 3 0 0 atd
Dec 15 19:02:18 teela kernel: [ 2628] 0 2628 26097 3193 27 3 0 0 NetworkManager
Dec 15 19:02:18 teela kernel: [ 2647] 0 2647 1511 458 5 3 0 0 fcron
Dec 15 19:02:18 teela kernel: [ 2673] 0 2673 750 543 6 3 0 0 dhcpcd
Dec 15 19:02:18 teela kernel: [ 2676] 0 2676 638 447 5 3 0 0 vnstatd
Dec 15 19:02:18 teela kernel: [ 2690] 0 2690 1457 1061 6 3 0 -1000 sshd
Dec 15 19:02:18 teela kernel: [ 2716] 106 2716 16384 4239 20 3 0 0 polkitd
Dec 15 19:02:18 teela kernel: [ 2717] 0 2717 2145 1360 7 3 0 0 wpa_supplicant
Dec 15 19:02:18 teela kernel: [ 2947] 0 2947 1794 775 7 3 0 0 screen
Dec 15 19:02:18 teela kernel: [ 2950] 0 2950 1831 913 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: [ 2954] 0 2954 37411 36100 79 3 0 0 emerge
Dec 15 19:02:18 teela kernel: [ 2970] 0 2970 1152 507 6 3 0 0 agetty
Dec 15 19:02:18 teela kernel: [ 3906] 250 3906 2625 1584 8 3 0 0 ebuild.sh
Dec 15 19:02:18 teela kernel: [ 3926] 250 3926 2657 1370 7 3 0 0 ebuild.sh
Dec 15 19:02:18 teela kernel: [ 3935] 250 3935 17160 16891 37 3 0 0 xz
Dec 15 19:02:18 teela kernel: [ 3936] 250 3936 799 510 5 3 0 0 tar
Dec 15 19:02:18 teela kernel: [ 4117] 0 4117 2598 1389 9 3 0 0 sshd
Dec 15 19:02:18 teela kernel: [ 4119] 0 4119 1964 1243 7 3 0 0 systemd
Dec 15 19:02:18 teela kernel: [ 4144] 0 4144 6645 632 10 3 0 0 (sd-pam)
Dec 15 19:02:18 teela kernel: [ 4163] 0 4163 1830 909 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: [ 4182] 0 4182 1695 684 7 3 0 0 screen
Dec 15 19:02:18 teela kernel: [ 4221] 0 4221 1831 893 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: Out of memory: Kill process 2954 (emerge) score 11 or sacrifice child
Dec 15 19:02:18 teela kernel: Killed process 2954 (emerge) total-vm:149644kB, anon-rss:137136kB, file-rss:7264kB, shmem-rss:0kB
Dec 15 19:02:18 teela kernel: bash invoked oom-killer: gfp_mask=0x27080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=0, order=1, oom_score_adj=0
Dec 15 19:02:18 teela kernel: bash cpuset=/ mems_allowed=0
Dec 15 19:02:18 teela kernel: CPU: 0 PID: 4221 Comm: bash Not tainted 4.9.0-gentoo #2
Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014
Dec 15 19:02:18 teela kernel: c5187d68 c142bcce c5187e98 00000000 c5187d98 c1163332 00000000 00000282
Dec 15 19:02:18 teela kernel: c5187d98 c1431876 c5187d9c e7f9e100 e7f947c0 e7f947c0 c1b58785 c5187e98
Dec 15 19:02:18 teela kernel: c5187ddc c110795f c1043895 c5187dc8 c11075c7 00000007 00000000 00000000
Dec 15 19:02:18 teela kernel: Call Trace:
Dec 15 19:02:18 teela kernel: [<c142bcce>] dump_stack+0x47/0x69
Dec 15 19:02:18 teela kernel: [<c1163332>] dump_header+0x60/0x178
Dec 15 19:02:18 teela kernel: [<c1431876>] ? ___ratelimit+0x86/0xe0
Dec 15 19:02:18 teela kernel: [<c110795f>] oom_kill_process+0x20f/0x3d0
Dec 15 19:02:18 teela kernel: [<c1043895>] ? has_capability_noaudit+0x15/0x20
Dec 15 19:02:18 teela kernel: [<c11075c7>] ? oom_badness.part.13+0xb7/0x130
Dec 15 19:02:18 teela kernel: [<c1107df9>] out_of_memory+0xd9/0x260
Dec 15 19:02:18 teela kernel: [<c110b989>] __alloc_pages_nodemask+0xb79/0xc80
Dec 15 19:02:18 teela kernel: [<c1038a05>] copy_process.part.51+0xe5/0x1420
Dec 15 19:02:18 teela kernel: [<c1150377>] ? kmem_cache_alloc+0xb7/0x190
Dec 15 19:02:18 teela kernel: [<c1039ee7>] _do_fork+0xc7/0x360
Dec 15 19:02:18 teela kernel: [<c1182a4b>] ? fd_install+0x1b/0x20
Dec 15 19:02:18 teela kernel: [<c103a247>] SyS_clone+0x27/0x30
Dec 15 19:02:18 teela kernel: [<c10018bc>] do_fast_syscall_32+0x7c/0x130
Dec 15 19:02:18 teela kernel: [<c19b5d2b>] sysenter_past_esp+0x40/0x6a
Dec 15 19:02:18 teela kernel: Mem-Info:
Dec 15 19:02:18 teela kernel: active_anon:22769 inactive_anon:90 isolated_anon:0
active_file:274396 inactive_file:281929 isolated_file:0
unevictable:0 dirty:616 writeback:0 unstable:0
slab_reclaimable:40669 slab_unreclaimable:17741
mapped:6595 shmem:202 pagetables:271 bounce:0
free:242474 free_pcp:608 free_cma:0
Dec 15 19:02:18 teela kernel: Node 0 active_anon:91076kB inactive_anon:360kB active_file:1097584kB inactive_file:1127716kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:26380kB dirty:2464kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 108544kB anon_thp: 808kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Dec 15 19:02:18 teela kernel: DMA free:3904kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:7356kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15916kB mlocked:0kB slab_reclaimable:3208kB slab_unreclaimable:1448kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 813 3474 3474
Dec 15 19:02:18 teela kernel: Normal free:41280kB min:41368kB low:51708kB high:62048kB active_anon:0kB inactive_anon:0kB active_file:532796kB inactive_file:44kB unevictable:0kB writepending:144kB present:897016kB managed:836248kB mlocked:0kB slab_reclaimable:159468kB slab_unreclaimable:69516kB kernel_stack:1104kB pagetables:1084kB bounce:0kB free_pcp:1048kB local_pcp:608kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 0 21292 21292
Dec 15 19:02:18 teela kernel: HighMem free:924712kB min:512kB low:34356kB high:68200kB active_anon:91076kB inactive_anon:360kB active_file:557432kB inactive_file:1127672kB unevictable:0kB writepending:2320kB present:2725384kB managed:2725384kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:1384kB local_pcp:656kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 0 0 0
Dec 15 19:02:18 teela kernel: DMA: 0*4kB 2*8kB (ME) 5*16kB (UME) 13*32kB (UM) 11*64kB (UME) 3*128kB (UM) 1*256kB (M) 2*512kB (E) 1*1024kB (M) 0*2048kB 0*4096kB = 3904kB
Dec 15 19:02:18 teela kernel: Normal: 26*4kB (M) 26*8kB (UM) 441*16kB (UM) 188*32kB (UM) 410*64kB (UME) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41288kB
Dec 15 19:02:18 teela kernel: HighMem: 1518*4kB (UM) 608*8kB (UM) 155*16kB (UM) 67*32kB (UM) 34*64kB (UM) 6*128kB (M) 2*256kB (M) 3*512kB (M) 1*1024kB (M) 43*2048kB (M) 199*4096kB (UM) = 924744kB
Dec 15 19:02:18 teela kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 15 19:02:18 teela kernel: 556527 total pagecache pages
Dec 15 19:02:18 teela kernel: 0 pages in swap cache
Dec 15 19:02:18 teela kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 15 19:02:18 teela kernel: Free swap = 8191996kB
Dec 15 19:02:18 teela kernel: Total swap = 8191996kB
Dec 15 19:02:18 teela kernel: 909598 pages RAM
Dec 15 19:02:18 teela kernel: 681346 pages HighMem/MovableOnly
Dec 15 19:02:18 teela kernel: 15211 pages reserved
Dec 15 19:02:18 teela kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 15 19:02:18 teela kernel: [ 1888] 0 1888 6161 1112 10 3 0 0 systemd-journal
Dec 15 19:02:18 teela kernel: [ 2508] 0 2508 2959 945 6 3 0 -1000 systemd-udevd
Dec 15 19:02:18 teela kernel: [ 2610] 105 2610 3870 899 8 3 0 0 systemd-timesyn
Dec 15 19:02:18 teela kernel: [ 2613] 0 2613 6300 951 10 3 0 0 rsyslogd
Dec 15 19:02:18 teela kernel: [ 2615] 88 2615 1158 568 6 3 0 0 nullmailer-send
Dec 15 19:02:18 teela kernel: [ 2618] 0 2618 1514 1027 7 3 0 0 systemd-logind
Dec 15 19:02:18 teela kernel: [ 2619] 101 2619 1266 847 6 3 0 -900 dbus-daemon
Dec 15 19:02:18 teela kernel: [ 2620] 0 2620 622 300 5 3 0 0 atd
Dec 15 19:02:18 teela kernel: [ 2628] 0 2628 26097 3193 27 3 0 0 NetworkManager
Dec 15 19:02:18 teela kernel: [ 2647] 0 2647 1511 458 5 3 0 0 fcron
Dec 15 19:02:18 teela kernel: [ 2673] 0 2673 750 543 6 3 0 0 dhcpcd
Dec 15 19:02:18 teela kernel: [ 2676] 0 2676 638 447 5 3 0 0 vnstatd
Dec 15 19:02:18 teela kernel: [ 2690] 0 2690 1457 1061 6 3 0 -1000 sshd
Dec 15 19:02:18 teela kernel: [ 2716] 106 2716 16384 4239 20 3 0 0 polkitd
Dec 15 19:02:18 teela kernel: [ 2717] 0 2717 2145 1360 7 3 0 0 wpa_supplicant
Dec 15 19:02:18 teela kernel: [ 2947] 0 2947 1794 775 7 3 0 0 screen
Dec 15 19:02:18 teela kernel: [ 2950] 0 2950 1831 915 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: [ 2970] 0 2970 1152 507 6 3 0 0 agetty
Dec 15 19:02:18 teela kernel: [ 3906] 250 3906 2625 1584 8 3 0 0 ebuild.sh
Dec 15 19:02:18 teela kernel: [ 3926] 250 3926 2657 1370 7 3 0 0 ebuild.sh
Dec 15 19:02:18 teela kernel: [ 3935] 250 3935 17160 16891 37 3 0 0 xz
Dec 15 19:02:18 teela kernel: [ 3936] 250 3936 799 510 5 3 0 0 tar
Dec 15 19:02:18 teela kernel: [ 4117] 0 4117 2598 1389 9 3 0 0 sshd
Dec 15 19:02:18 teela kernel: [ 4119] 0 4119 1964 1243 7 3 0 0 systemd
Dec 15 19:02:18 teela kernel: [ 4144] 0 4144 6645 632 10 3 0 0 (sd-pam)
Dec 15 19:02:18 teela kernel: [ 4163] 0 4163 1830 909 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: [ 4182] 0 4182 1695 684 7 3 0 0 screen
Dec 15 19:02:18 teela kernel: [ 4221] 0 4221 1831 893 7 3 0 0 bash
Dec 15 19:02:18 teela kernel: Out of memory: Kill process 3935 (xz) score 5 or sacrifice child
Dec 15 19:02:18 teela kernel: Killed process 3935 (xz) total-vm:68640kB, anon-rss:65928kB, file-rss:1636kB, shmem-rss:0kB
Dec 15 19:02:18 teela kernel: ebuild.sh invoked oom-killer: gfp_mask=0x27080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO|__GFP_NOTRACK), nodemask=0, order=1, oom_score_adj=0
Dec 15 19:02:18 teela kernel: ebuild.sh cpuset=/ mems_allowed=0
Dec 15 19:02:18 teela kernel: CPU: 0 PID: 3926 Comm: ebuild.sh Not tainted 4.9.0-gentoo #2
Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014
Dec 15 19:02:18 teela kernel: d3473d68 c142bcce d3473e98 00000000 d3473d98 c1163332 00000000 00000282
Dec 15 19:02:18 teela kernel: d3473d98 c1431876 d3473d9c f12463c0 f25ea900 f25ea900 c1b58785 d3473e98
Dec 15 19:02:18 teela kernel: d3473ddc c110795f c1043895 d3473dc8 c11075c7 00000006 00000000 00000000
Dec 15 19:02:18 teela kernel: Call Trace:
Dec 15 19:02:18 teela kernel: [<c142bcce>] dump_stack+0x47/0x69
Dec 15 19:02:18 teela kernel: [<c1163332>] dump_header+0x60/0x178
Dec 15 19:02:18 teela kernel: [<c1431876>] ? ___ratelimit+0x86/0xe0
Dec 15 19:02:18 teela kernel: [<c110795f>] oom_kill_process+0x20f/0x3d0
Dec 15 19:02:18 teela kernel: [<c1043895>] ? has_capability_noaudit+0x15/0x20
Dec 15 19:02:18 teela kernel: [<c11075c7>] ? oom_badness.part.13+0xb7/0x130
Dec 15 19:02:18 teela kernel: [<c1107df9>] out_of_memory+0xd9/0x260
Dec 15 19:02:18 teela kernel: [<c110b989>] __alloc_pages_nodemask+0xb79/0xc80
Dec 15 19:02:18 teela kernel: [<c1151a00>] ? __kmem_cache_shutdown+0x220/0x290
Dec 15 19:02:18 teela kernel: [<c1038a05>] copy_process.part.51+0xe5/0x1420
Dec 15 19:02:18 teela kernel: [<c1150377>] ? kmem_cache_alloc+0xb7/0x190
Dec 15 19:02:18 teela kernel: [<c1039ee7>] _do_fork+0xc7/0x360
Dec 15 19:02:18 teela kernel: [<c1438c64>] ? _copy_to_user+0x44/0x60
Dec 15 19:02:18 teela kernel: [<c103a247>] SyS_clone+0x27/0x30
Dec 15 19:02:18 teela kernel: [<c10018bc>] do_fast_syscall_32+0x7c/0x130
Dec 15 19:02:18 teela kernel: [<c19b5d2b>] sysenter_past_esp+0x40/0x6a
Dec 15 19:02:18 teela kernel: Mem-Info:
Dec 15 19:02:18 teela kernel: active_anon:6238 inactive_anon:90 isolated_anon:0
active_file:274469 inactive_file:281903 isolated_file:0
unevictable:0 dirty:557 writeback:255 unstable:0
slab_reclaimable:40673 slab_unreclaimable:17738
mapped:6479 shmem:202 pagetables:238 bounce:0
free:258997 free_pcp:617 free_cma:0
Dec 15 19:02:18 teela kernel: Node 0 active_anon:24952kB inactive_anon:360kB active_file:1097876kB inactive_file:1127612kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:25916kB dirty:2228kB writeback:1020kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 6144kB anon_thp: 808kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Dec 15 19:02:18 teela kernel: DMA free:3904kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:7356kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15916kB mlocked:0kB slab_reclaimable:3208kB slab_unreclaimable:1448kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 813 3474 3474
Dec 15 19:02:18 teela kernel: Normal free:41272kB min:41368kB low:51708kB high:62048kB active_anon:0kB inactive_anon:0kB active_file:532844kB inactive_file:48kB unevictable:0kB writepending:400kB present:897016kB managed:836248kB mlocked:0kB slab_reclaimable:159484kB slab_unreclaimable:69504kB kernel_stack:1096kB pagetables:952kB bounce:0kB free_pcp:1128kB local_pcp:588kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 0 21292 21292
Dec 15 19:02:18 teela kernel: HighMem free:990812kB min:512kB low:34356kB high:68200kB active_anon:24952kB inactive_anon:360kB active_file:557676kB inactive_file:1127564kB unevictable:0kB writepending:2848kB present:2725384kB managed:2725384kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:1340kB local_pcp:688kB free_cma:0kB
Dec 15 19:02:18 teela kernel: lowmem_reserve[]: 0 0 0 0
Dec 15 19:02:18 teela kernel: DMA: 0*4kB 2*8kB (ME) 5*16kB (UME) 13*32kB (UM) 11*64kB (UME) 3*128kB (UM) 1*256kB (M) 2*512kB (E) 1*1024kB (M) 0*2048kB 0*4096kB = 3904kB
Dec 15 19:02:18 teela kernel: Normal: 30*4kB (UME) 31*8kB (UM) 437*16kB (UME) 188*32kB (UM) 410*64kB (UME) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 41280kB
Dec 15 19:02:18 teela kernel: HighMem: 1621*4kB (UM) 660*8kB (UM) 184*16kB (UM) 90*32kB (UM) 41*64kB (UM) 7*128kB (M) 2*256kB (M) 3*512kB (M) 1*1024kB (M) 50*2048kB (M) 211*4096kB (UM) = 990836kB
Dec 15 19:02:18 teela kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 15 19:02:18 teela kernel: 556574 total pagecache pages
Dec 15 19:02:18 teela kernel: 0 pages in swap cache
Dec 15 19:02:18 teela kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 15 19:02:18 teela kernel: Free swap = 8191996kB
Dec 15 19:02:18 teela kernel: Total swap = 8191996kB
Dec 15 19:02:18 teela kernel: 909598 pages RAM
Dec 15 19:02:18 teela kernel: 681346 pages HighMem/MovableOnly
Dec 15 19:02:18 teela kernel: 15211 pages reserved
Dec 15 19:02:18 teela kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 15 19:02:18 teela kernel: [ 1888] 0 1888 6161 1143 10 3 0 0 systemd-journal
Dec 15 19:02:18 teela kernel: [ 2508] 0 2508 2959 945 6 3 0 -1000 systemd-udevd
Dec 15 19:02:18 teela kernel: [ 2610] 105 2610 3870 899 8 3 0 0 systemd-timesyn
Dec 15 19:02:18 teela kernel: [ 2613] 0 2613 6300 956 10 3 0 0 rsyslogd
Dec 15 19:02:18 teela kernel: [ 2615] 88 2615 1158 568 6 3 0 0 nullmailer-send
Dec 15 19:02:18 teela kernel: [ 2618] 0 2618 1514 1027 7 3 0 0 systemd-logind
Dec 15 19:02:18 teela kernel: [ 2619] 101 2619 1266 847 6 3 0 -900 dbus-daemon
Dec 15 19:02:18 teela kernel: [ 2620] 0 2620 622 300 5 3 0 0 atd
Dec 15 19:02:26 teela kernel: [ 2628] 0 2628 26097 3193 27 3 0 0 NetworkManager
Dec 15 19:02:26 teela kernel: [ 2647] 0 2647 1511 458 5 3 0 0 fcron
Dec 15 19:02:26 teela kernel: [ 2673] 0 2673 750 543 6 3 0 0 dhcpcd
Dec 15 19:02:26 teela kernel: [ 2676] 0 2676 638 447 5 3 0 0 vnstatd
Dec 15 19:02:26 teela kernel: [ 2690] 0 2690 1457 1061 6 3 0 -1000 sshd
Dec 15 19:02:26 teela kernel: [ 2716] 106 2716 16384 4239 20 3 0 0 polkitd
Dec 15 19:02:26 teela kernel: [ 2717] 0 2717 2145 1360 7 3 0 0 wpa_supplicant
Dec 15 19:02:26 teela kernel: [ 2947] 0 2947 1794 775 7 3 0 0 screen
Dec 15 19:02:26 teela kernel: [ 2950] 0 2950 1831 915 7 3 0 0 bash
Dec 15 19:02:26 teela kernel: [ 2970] 0 2970 1152 507 6 3 0 0 agetty
Dec 15 19:02:26 teela kernel: [ 3906] 250 3906 2625 1584 8 3 0 0 ebuild.sh
Dec 15 19:02:26 teela kernel: [ 3926] 250 3926 2657 1377 7 3 0 0 ebuild.sh
Dec 15 19:02:26 teela kernel: [ 4117] 0 4117 2598 1389 9 3 0 0 sshd
Dec 15 19:02:26 teela kernel: [ 4119] 0 4119 1964 1243 7 3 0 0 systemd
Dec 15 19:02:26 teela kernel: [ 4144] 0 4144 6645 632 10 3 0 0 (sd-pam)
Dec 15 19:02:26 teela kernel: [ 4163] 0 4163 1830 909 7 3 0 0 bash
Dec 15 19:02:26 teela kernel: [ 4182] 0 4182 1695 684 7 3 0 0 screen
Dec 15 19:02:26 teela kernel: [ 4221] 0 4221 1831 893 7 3 0 0 bash
Dec 15 19:02:26 teela kernel: [ 4225] 0 4225 1831 400 6 3 0 0 bash
Greetings
Nils