3.4-rc4 oom killer out of control.
From: Dave Jones
Date: Thu Apr 26 2012 - 15:35:56 EST
On a test machine that was running my system call fuzzer, I just saw
the oom killer take out everything but the process that was doing all
the memory exhausting.
Partial logs below. The machine locked up completely (even capslock wouldn't work).
The console had logs up to 5041.xxxxxx before the wedge, but they never made it to disk.
It was just more of the same below..
Note that the trinity processes have largest RSS, yet seem immune to getting killed.
Dave
Apr 26 13:56:45 dhcp-189-232 kernel: [ 4959.198003] modprobe invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.198349] modprobe cpuset=/ mems_allowed=0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.198705] Pid: 14950, comm: modprobe Not tainted 3.4.0-rc4+ #55
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.199406] Call Trace:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.200214] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.201084] [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.201928] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.202814] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.203687] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.204539] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.205416] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.206277] [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.207142] [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.207999] [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.208892] [<ffffffff8117df8f>] __do_fault+0x6f/0x540
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.209752] [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.210633] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.211509] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.212384] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.213475] [<ffffffff81335a1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.214137] [<ffffffff816af019>] ? retint_swapgs+0x13/0x1b
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.215018] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.215886] [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.216722] Mem-Info:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.217608] Node 0 DMA per-cpu:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.218500] CPU 0: hi: 0, btch: 1 usd: 0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.219400] CPU 1: hi: 0, btch: 1 usd: 0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.220292] CPU 2: hi: 0, btch: 1 usd: 0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.221186] CPU 3: hi: 0, btch: 1 usd: 0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.222067] Node 0 DMA32 per-cpu:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.222937] CPU 0: hi: 186, btch: 31 usd: 16
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.223833] CPU 1: hi: 186, btch: 31 usd: 7
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.224692] CPU 2: hi: 186, btch: 31 usd: 1
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.225554] CPU 3: hi: 186, btch: 31 usd: 9
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.226408] Node 0 Normal per-cpu:
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.227253] CPU 0: hi: 186, btch: 31 usd: 2
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.228095] CPU 1: hi: 186, btch: 31 usd: 65
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.228918] CPU 2: hi: 186, btch: 31 usd: 33
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.229767] CPU 3: hi: 186, btch: 31 usd: 21
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230600] active_anon:1421942 inactive_anon:313560 isolated_anon:0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230604] active_file:88 inactive_file:120 isolated_file:70
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230608] unevictable:3038 dirty:7 writeback:0 unstable:0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230610] free:25574 slab_reclaimable:14238 slab_unreclaimable:101544
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.230611] mapped:232 shmem:86 pagetables:89711 bounce:0
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.234745] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.237548] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:56:48 dhcp-189-232 kernel: [ 4959.238522] Node 0 DMA32 free:46340kB min:27252kB low:34064kB high:40876kB active_anon:2330464kB inactive_anon:582676kB active_file:16kB inactive_file:44kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:112kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132232kB kernel_stack:2272kB pagetables:133664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:42 all_unreclaimable? yes
Apr 26 13:56:50 dhcp-189-232 kernel: [ 4959.241635] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:09 dhcp-189-232 kernel: [ 4959.242701] Node 0 Normal free:40080kB min:40196kB low:50244kB high:60292kB active_anon:3357304kB inactive_anon:671564kB active_file:336kB inactive_file:436kB unevictable:7404kB isolated(anon):0kB isolated(file):280kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:816kB shmem:240kB slab_reclaimable:54120kB slab_unreclaimable:273912kB kernel_stack:4528kB pagetables:225180kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:970 all_unreclaimable? yes
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.246216] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.247423] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.248684] Node 0 DMA32: 217*4kB 152*8kB 158*16kB 232*32kB 134*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46340kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.249873] Node 0 Normal: 475*4kB 405*8kB 272*16kB 352*32kB 144*64kB 40*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 39956kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.251173] 2037 total pagecache pages
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.252403] 1692 pages in swap cache
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.253630] Swap cache stats: add 1828870, delete 1827178, find 493465/493919
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.254887] Free swap = 0kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.256126] Total swap = 1023996kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.347525] 2097136 pages RAM
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.348055] 59776 pages reserved
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.348865] 2554149 pages shared
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.349991] 2007673 pages non-shared
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.351168] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.352382] [ 353] 0 353 4425 23 2 -17 -1000 udevd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.353575] [ 355] 0 355 5802 1 2 0 0 systemd-stdout-
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.354796] [ 737] 0 737 4766 1 3 0 0 smartd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.356016] [ 751] 0 751 29591 24 0 0 0 crond
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.357246] [ 755] 0 755 1616 1 1 0 0 acpid
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.358447] [ 756] 0 756 67222 107 0 0 0 NetworkManager
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.359656] [ 770] 0 770 7083 1 0 0 0 systemd-logind
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.360879] [ 773] 70 773 7005 50 3 0 0 avahi-daemon
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.362082] [ 776] 70 776 6972 4 0 0 0 avahi-daemon
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.363301] [ 778] 0 778 1612 1 0 0 0 mcelog
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.364526] [ 783] 994 783 4893 24 0 0 0 chronyd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.365748] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.366862] [ 790] 0 790 29064 39 0 0 0 ksmtuned
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.368085] [ 797] 0 797 46610 31 1 -13 -900 polkitd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.369305] [ 802] 0 802 60782 52 1 0 0 rsyslogd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.370527] [ 815] 0 815 18861 1 0 -17 -1000 sshd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.371756] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.372995] [ 819] 0 819 4792 17 2 0 0 rpcbind
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.374229] [ 824] 29 824 6938 1 0 0 0 rpc.statd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.375462] [ 837] 0 837 27232 1 3 0 0 agetty
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.376695] [ 842] 0 842 21549 1 2 0 0 dhclient
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.377938] [ 1170] 1000 1170 6134 81 0 0 0 tmux
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.379192] [ 1171] 1000 1171 29914 1 2 0 0 bash
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.380455] [ 1262] 1000 1262 4022 125 3 0 0 trinity
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.381709] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.382928] [15065] 0 15065 3725 87 3 0 0 anacron
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.384164] [19925] 0 19925 4424 1 3 -17 -1000 udevd
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.385449] [13786] 0 13786 26704 68 2 0 0 sleep
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.386748] [14941] 0 14941 1629 87 0 0 0 modprobe
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.388042] [14944] 0 14944 1628 63 0 0 0 modprobe
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.389325] [14945] 1000 14945 4418 639 3 0 0 trinity
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.390618] [14950] 0 14950 106 3 2 0 0 modprobe
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.391913] [14952] 1000 14952 4022 203 0 0 0 trinity
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.393252] [14955] 1000 14955 4022 124 0 0 0 trinity
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.394560] Out of memory: Kill process 355 (systemd-stdout-) score 1 or sacrifice child
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4959.395847] Killed process 355 (systemd-stdout-) total-vm:23208kB, anon-rss:0kB, file-rss:4kB
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.358425] systemd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.358956] systemd cpuset=/ mems_allowed=0
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.359956] Pid: 1, comm: systemd Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:10 dhcp-189-232 kernel: [ 4961.361271] Call Trace:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.362557] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.363827] [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.365052] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.366208] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.367393] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.368532] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.369673] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.370780] [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.371855] [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.372910] [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.373935] [<ffffffff8117df8f>] __do_fault+0x6f/0x540
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.374939] [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.375917] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.376883] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.377826] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.378765] [<ffffffff8120f046>] ? sys_epoll_wait+0x96/0x470
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.379700] [<ffffffff816b6b15>] ? sysret_check+0x22/0x5d
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.380620] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.381549] [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.382409] Mem-Info:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.383329] Node 0 DMA per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.384259] CPU 0: hi: 0, btch: 1 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.385200] CPU 1: hi: 0, btch: 1 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.386123] CPU 2: hi: 0, btch: 1 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.387024] CPU 3: hi: 0, btch: 1 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.387898] Node 0 DMA32 per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.388802] CPU 0: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.389691] CPU 1: hi: 186, btch: 31 usd: 30
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.390587] CPU 2: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.391466] CPU 3: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.392334] Node 0 Normal per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.393198] CPU 0: hi: 186, btch: 31 usd: 75
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.394078] CPU 1: hi: 186, btch: 31 usd: 30
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.394934] CPU 2: hi: 186, btch: 31 usd: 22
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.395810] CPU 3: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396663] active_anon:1421870 inactive_anon:313545 isolated_anon:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396664] active_file:103 inactive_file:50 isolated_file:64
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396665] unevictable:3038 dirty:7 writeback:0 unstable:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396666] free:25649 slab_reclaimable:14238 slab_unreclaimable:101538
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.396668] mapped:237 shmem:86 pagetables:89658 bounce:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.400859] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.403671] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.404653] Node 0 DMA32 free:46192kB min:27252kB low:34064kB high:40876kB active_anon:2330460kB inactive_anon:582672kB active_file:0kB inactive_file:96kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:108kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132232kB kernel_stack:2272kB pagetables:133664kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.407764] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.408824] Node 0 Normal free:40528kB min:40196kB low:50244kB high:60292kB active_anon:3357020kB inactive_anon:671508kB active_file:412kB inactive_file:104kB unevictable:7404kB isolated(anon):0kB isolated(file):256kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:840kB shmem:240kB slab_reclaimable:54120kB slab_unreclaimable:273888kB kernel_stack:4528kB pagetables:224968kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:23 all_unreclaimable? no
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.412250] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.413420] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.414664] Node 0 DMA32: 226*4kB 155*8kB 160*16kB 229*32kB 134*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46336kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.415839] Node 0 Normal: 542*4kB 428*8kB 273*16kB 353*32kB 145*64kB 40*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40520kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.417112] 2026 total pagecache pages
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.418352] 1685 pages in swap cache
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.419591] Swap cache stats: add 1828918, delete 1827233, find 493469/493923
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.420839] Free swap = 0kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.422068] Total swap = 1023996kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.502902] 2097136 pages RAM
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.503469] 59776 pages reserved
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.504253] 2553433 pages shared
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.505446] 2007548 pages non-shared
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.506614] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.507848] [ 353] 0 353 4425 23 2 -17 -1000 udevd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.509055] [ 737] 0 737 4766 1 3 0 0 smartd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.510247] [ 751] 0 751 29591 24 0 0 0 crond
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.511462] [ 755] 0 755 1616 1 1 0 0 acpid
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.512665] [ 756] 0 756 67222 107 1 0 0 NetworkManager
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.513879] [ 770] 0 770 7083 1 0 0 0 systemd-logind
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.515077] [ 773] 70 773 7005 50 3 0 0 avahi-daemon
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.516211] [ 776] 70 776 6972 4 0 0 0 avahi-daemon
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.517431] [ 778] 0 778 1612 1 0 0 0 mcelog
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.518630] [ 783] 994 783 4893 24 0 0 0 chronyd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.519848] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.521071] [ 790] 0 790 29064 39 0 0 0 ksmtuned
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.522285] [ 797] 0 797 46610 31 2 -13 -900 polkitd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.523503] [ 802] 0 802 60782 52 1 0 0 rsyslogd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.524745] [ 815] 0 815 18861 1 0 -17 -1000 sshd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.525972] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.527201] [ 819] 0 819 4792 17 2 0 0 rpcbind
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.528428] [ 824] 29 824 6938 1 0 0 0 rpc.statd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.529659] [ 837] 0 837 27232 1 3 0 0 agetty
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.530899] [ 842] 0 842 21549 1 2 0 0 dhclient
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.532097] [ 1170] 1000 1170 6134 78 2 0 0 tmux
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.533306] [ 1171] 1000 1171 29914 1 2 0 0 bash
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.534547] [ 1262] 1000 1262 4022 125 1 0 0 trinity
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.535816] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.537076] [15065] 0 15065 3725 87 3 0 0 anacron
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.538337] [19925] 0 19925 4424 1 3 -17 -1000 udevd
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.539605] [13786] 0 13786 26704 68 2 0 0 sleep
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.540888] [14941] 0 14941 1629 87 1 0 0 modprobe
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.542179] [14944] 0 14944 1628 63 1 0 0 modprobe
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.543473] [14945] 1000 14945 4418 652 2 0 0 trinity
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.544756] [14950] 0 14950 106 7 3 0 0 modprobe
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.546051] [14955] 1000 14955 4022 124 2 0 0 trinity
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.547353] Out of memory: Kill process 737 (smartd) score 1 or sacrifice child
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4961.548644] Killed process 737 (smartd) total-vm:19064kB, anon-rss:0kB, file-rss:4kB
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.791375] trinity invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.791871] trinity cpuset=/ mems_allowed=0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.792948] Pid: 14972, comm: trinity Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.794230] Call Trace:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.795559] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.796866] [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.798155] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.799432] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.800679] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.801892] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.803092] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.804244] [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.805393] [<ffffffff81181300>] handle_pte_fault+0x6a0/0xa10
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.806510] [<ffffffff810a2981>] ? get_parent_ip+0x11/0x50
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807590] [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807596] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807602] [<ffffffff811b4f68>] do_huge_pmd_anonymous_page+0xc8/0x380
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807607] [<ffffffff81181976>] handle_mm_fault+0x146/0x2f0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807610] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807614] [<ffffffff811871d6>] ? do_brk+0x246/0x360
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807619] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807623] [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807627] Mem-Info:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807630] Node 0 DMA per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807633] CPU 0: hi: 0, btch: 1 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807637] CPU 1: hi: 0, btch: 1 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807639] CPU 2: hi: 0, btch: 1 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807641] CPU 3: hi: 0, btch: 1 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807643] Node 0 DMA32 per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807646] CPU 0: hi: 186, btch: 31 usd: 1
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807648] CPU 1: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807651] CPU 2: hi: 186, btch: 31 usd: 3
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807653] CPU 3: hi: 186, btch: 31 usd: 1
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807655] Node 0 Normal per-cpu:
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807658] CPU 0: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807660] CPU 1: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807662] CPU 2: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807664] CPU 3: hi: 186, btch: 31 usd: 0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807670] active_anon:1421827 inactive_anon:313613 isolated_anon:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807672] active_file:63 inactive_file:73 isolated_file:110
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807673] unevictable:3038 dirty:7 writeback:0 unstable:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807674] free:25681 slab_reclaimable:14174 slab_unreclaimable:101572
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807675] mapped:229 shmem:86 pagetables:89725 bounce:0
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807677] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807687] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807692] Node 0 DMA32 free:46508kB min:27252kB low:34064kB high:40876kB active_anon:2330216kB inactive_anon:582636kB active_file:24kB inactive_file:28kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2296kB pagetables:133680kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:83 all_unreclaimable? yes
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807702] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:14 dhcp-189-232 systemd[1]: systemd-stdout-syslog-bridge.service: main process exited, code=killed, status=9
Apr 26 13:57:11 dhcp-189-232 kernel: [ 4975.807706] Node 0 Normal free:40340kB min:40196kB low:50244kB high:60292kB active_anon:3357092kB inactive_anon:671816kB active_file:228kB inactive_file:264kB unevictable:7404kB isolated(anon):0kB isolated(file):440kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4576kB pagetables:225220kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:770 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807717] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807720] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807731] Node 0 DMA32: 219*4kB 156*8kB 166*16kB 230*32kB 135*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46508kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807742] Node 0 Normal: 517*4kB 412*8kB 266*16kB 350*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40340kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807752] 1526 total pagecache pages
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807754] 1249 pages in swap cache
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807757] Swap cache stats: add 1829703, delete 1828454, find 493950/494458
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807759] Free swap = 0kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.807760] Total swap = 1023996kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.924517] 2097136 pages RAM
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.925073] 59776 pages reserved
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.925829] 2551230 pages shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.927017] 2007729 pages non-shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.928103] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.929299] [ 353] 0 353 4425 23 2 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.930485] [ 751] 0 751 29591 24 0 0 0 crond
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.931669] [ 755] 0 755 1616 1 1 0 0 acpid
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.932860] [ 756] 0 756 67222 107 1 0 0 NetworkManager
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.934065] [ 770] 0 770 7083 1 0 0 0 systemd-logind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.935251] [ 773] 70 773 7005 50 3 0 0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.936454] [ 776] 70 776 6972 4 0 0 0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.937669] [ 778] 0 778 1612 1 0 0 0 mcelog
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.938838] [ 783] 994 783 4893 24 0 0 0 chronyd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.940037] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.941223] [ 790] 0 790 29064 39 0 0 0 ksmtuned
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.942428] [ 797] 0 797 46610 31 0 -13 -900 polkitd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.943620] [ 802] 0 802 60782 59 1 0 0 rsyslogd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.944740] [ 815] 0 815 18861 1 0 -17 -1000 sshd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.945960] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.947173] [ 819] 0 819 4792 17 3 0 0 rpcbind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.948373] [ 824] 29 824 6938 1 0 0 0 rpc.statd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.949594] [ 837] 0 837 27232 1 3 0 0 agetty
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.950802] [ 842] 0 842 21549 1 2 0 0 dhclient
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.952021] [ 1170] 1000 1170 6133 77 0 0 0 tmux
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.953241] [ 1171] 1000 1171 29914 1 2 0 0 bash
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.954555] [ 1262] 1000 1262 4022 125 2 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.955715] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.956971] [15065] 0 15065 3725 87 3 0 0 anacron
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.958231] [19925] 0 19925 4424 1 3 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.959476] [14944] 0 14944 1629 86 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.960698] [14950] 0 14950 1595 57 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.961949] [14954] 0 14954 1595 36 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.963230] [14958] 0 14958 2810 19 2 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.964494] [14961] 1000 14961 4286 459 1 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.965773] [14965] 0 14965 1618 15 0 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.967069] [14966] 0 14966 2290 19 1 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.968363] [14967] 0 14967 1621 35 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.969669] [14970] 0 14970 1595 58 0 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.970972] [14972] 1000 14972 4220 456 3 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.972269] [14975] 0 14975 104 1 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.973563] Out of memory: Kill process 751 (crond) score 1 or sacrifice child
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4975.974863] Killed process 751 (crond) total-vm:118364kB, anon-rss:92kB, file-rss:4kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.036332] systemd-cgroups invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.036811] systemd-cgroups cpuset=/ mems_allowed=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.037772] Pid: 14958, comm: systemd-cgroups Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.039032] Call Trace:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.040280] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.041507] [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.042610] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.043793] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.044947] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.046063] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.047156] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.048226] [<ffffffff8119d366>] alloc_pages_current+0xb6/0x120
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.049259] [<ffffffff811569e7>] __page_cache_alloc+0xb7/0xe0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.050282] [<ffffffff81159bbf>] filemap_fault+0x2ff/0x4c0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.051270] [<ffffffff8117df8f>] __do_fault+0x6f/0x540
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.052234] [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.053175] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.054132] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.055081] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.056015] [<ffffffff811c1453>] ? sys_close+0x43/0x1a0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.056940] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.057895] [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.058855] Mem-Info:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.059727] Node 0 DMA per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.060674] CPU 0: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.061619] CPU 1: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.062547] CPU 2: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.063456] CPU 3: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.064373] Node 0 DMA32 per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.065269] CPU 0: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.066178] CPU 1: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.067069] CPU 2: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.067947] CPU 3: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.068854] Node 0 Normal per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.069730] CPU 0: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.070587] CPU 1: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.071458] CPU 2: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.072321] CPU 3: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073165] active_anon:1421858 inactive_anon:313540 isolated_anon:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073166] active_file:94 inactive_file:65 isolated_file:151
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073167] unevictable:3038 dirty:9 writeback:0 unstable:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073170] free:25544 slab_reclaimable:14174 slab_unreclaimable:101572
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.073172] mapped:229 shmem:86 pagetables:89708 bounce:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.077353] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.080149] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.081123] Node 0 DMA32 free:46240kB min:27252kB low:34064kB high:40876kB active_anon:2330228kB inactive_anon:582636kB active_file:28kB inactive_file:36kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:8kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2288kB pagetables:133684kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:5 all_unreclaimable? no
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.084201] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.085253] Node 0 Normal free:40060kB min:40196kB low:50244kB high:60292kB active_anon:3357204kB inactive_anon:671524kB active_file:348kB inactive_file:224kB unevictable:7404kB isolated(anon):0kB isolated(file):604kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4600kB pagetables:225148kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? no
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.088669] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.089845] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.091098] Node 0 DMA32: 204*4kB 143*8kB 161*16kB 231*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46360kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.092319] Node 0 Normal: 565*4kB 415*8kB 265*16kB 350*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40540kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.093613] 1616 total pagecache pages
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.094829] 1336 pages in swap cache
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.096071] Swap cache stats: add 1829817, delete 1828481, find 493950/494458
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.097332] Free swap = 0kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.098596] Total swap = 1023996kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.205067] 2097136 pages RAM
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.205470] 59776 pages reserved
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206413] 2549841 pages shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206415] 2007690 pages non-shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206418] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206444] [ 353] 0 353 4425 23 2 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206459] [ 755] 0 755 1616 1 1 0 0 acpid
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206465] [ 756] 0 756 67222 107 1 0 0 NetworkManager
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206470] [ 770] 0 770 7083 1 0 0 0 systemd-logind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206475] [ 773] 70 773 7005 50 3 0 0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206480] [ 776] 70 776 6972 4 0 0 0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206485] [ 778] 0 778 1612 1 0 0 0 mcelog
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206490] [ 783] 994 783 4893 24 0 0 0 chronyd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206495] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206500] [ 790] 0 790 29064 39 2 0 0 ksmtuned
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206505] [ 797] 0 797 46610 31 1 -13 -900 polkitd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206510] [ 802] 0 802 60782 59 1 0 0 rsyslogd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206515] [ 815] 0 815 18861 1 0 -17 -1000 sshd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206520] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206524] [ 819] 0 819 4792 17 3 0 0 rpcbind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206529] [ 824] 29 824 6938 1 0 0 0 rpc.statd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206534] [ 837] 0 837 27232 1 3 0 0 agetty
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206539] [ 842] 0 842 21549 1 2 0 0 dhclient
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206544] [ 1170] 1000 1170 6133 77 2 0 0 tmux
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206549] [ 1171] 1000 1171 29914 1 2 0 0 bash
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206554] [ 1262] 1000 1262 4022 125 3 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206559] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206565] [15065] 0 15065 3725 87 3 0 0 anacron
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206573] [19925] 0 19925 4424 1 3 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206581] [14944] 0 14944 1629 86 3 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206585] [14950] 0 14950 1628 63 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206590] [14954] 0 14954 1595 36 3 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206596] [14958] 0 14958 2810 20 0 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206600] [14961] 1000 14961 4286 459 1 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206606] [14965] 0 14965 1618 15 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206610] [14966] 0 14966 2290 19 3 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206615] [14967] 0 14967 1621 35 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206620] [14970] 0 14970 1628 62 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206625] [14972] 1000 14972 4286 529 3 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206631] [14975] 0 14975 106 8 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206636] Out of memory: Kill process 755 (acpid) score 1 or sacrifice child
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4981.206640] Killed process 755 (acpid) total-vm:6464kB, anon-rss:0kB, file-rss:4kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.882053] trinity invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.882827] trinity cpuset=/ mems_allowed=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.883513] Pid: 14972, comm: trinity Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.884768] Call Trace:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.886025] [<ffffffff816ae775>] ? _raw_spin_unlock+0x55/0x60
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.887232] [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.888433] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.889579] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.890703] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.891787] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.892873] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.893940] [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.894969] [<ffffffff81181300>] handle_pte_fault+0x6a0/0xa10
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.895959] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.896952] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.897917] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.898874] [<ffffffff811871d6>] ? do_brk+0x246/0x360
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.899797] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.900733] [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.901645] Mem-Info:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.902542] Node 0 DMA per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.903447] CPU 0: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.904352] CPU 1: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.905264] CPU 2: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.906166] CPU 3: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.907061] Node 0 DMA32 per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.907881] CPU 0: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.908779] CPU 1: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.909655] CPU 2: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.910525] CPU 3: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.911379] Node 0 Normal per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.912233] CPU 0: hi: 186, btch: 31 usd: 1
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.913091] CPU 1: hi: 186, btch: 31 usd: 4
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.913930] CPU 2: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.914792] CPU 3: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915639] active_anon:1421866 inactive_anon:313517 isolated_anon:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915641] active_file:86 inactive_file:127 isolated_file:114
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915642] unevictable:3038 dirty:9 writeback:0 unstable:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915643] free:25566 slab_reclaimable:14174 slab_unreclaimable:101572
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.915644] mapped:229 shmem:86 pagetables:89704 bounce:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.919846] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.922609] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.923570] Node 0 DMA32 free:46384kB min:27252kB low:34064kB high:40876kB active_anon:2330352kB inactive_anon:582628kB active_file:0kB inactive_file:64kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:8kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2832kB slab_unreclaimable:132256kB kernel_stack:2280kB pagetables:133684kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:228 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.926571] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.927617] Node 0 Normal free:40004kB min:40196kB low:50244kB high:60292kB active_anon:3357112kB inactive_anon:671440kB active_file:344kB inactive_file:444kB unevictable:7404kB isolated(anon):0kB isolated(file):456kB present:4902912kB mlocked:7404kB dirty:28kB writeback:0kB mapped:812kB shmem:240kB slab_reclaimable:53864kB slab_unreclaimable:274000kB kernel_stack:4600kB pagetables:225132kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12 all_unreclaimable? no
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.931017] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.932183] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.933420] Node 0 DMA32: 206*4kB 145*8kB 161*16kB 231*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46384kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.934698] Node 0 Normal: 506*4kB 408*8kB 263*16kB 351*32kB 145*64kB 42*128kB 11*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 40248kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.935988] 1630 total pagecache pages
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.937219] 1286 pages in swap cache
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.938456] Swap cache stats: add 1829855, delete 1828569, find 493952/494460
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.939701] Free swap = 0kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4985.940881] Total swap = 1023996kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.024043] 2097136 pages RAM
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.024445] 59776 pages reserved
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.025432] 2548364 pages shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.026651] 2007726 pages non-shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.027872] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.029123] [ 353] 0 353 4425 23 2 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.030346] [ 756] 0 756 67222 111 0 0 0 NetworkManager
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.031539] [ 770] 0 770 7083 1 0 0 0 systemd-logind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.032769] [ 773] 70 773 7005 50 3 0 0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.034004] [ 776] 70 776 6972 4 0 0 0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.035225] [ 778] 0 778 1612 1 0 0 0 mcelog
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.036452] [ 783] 994 783 4893 24 0 0 0 chronyd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.037654] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.038878] [ 790] 0 790 29064 39 1 0 0 ksmtuned
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.040105] [ 797] 0 797 46610 39 2 -13 -900 polkitd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.041252] [ 802] 0 802 60782 59 1 0 0 rsyslogd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.042478] [ 815] 0 815 18861 1 0 -17 -1000 sshd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.043706] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.044954] [ 819] 0 819 4792 17 3 0 0 rpcbind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.046187] [ 824] 29 824 6938 1 0 0 0 rpc.statd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.047426] [ 837] 0 837 27232 1 3 0 0 agetty
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.048669] [ 842] 0 842 21549 1 2 0 0 dhclient
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.049899] [ 1170] 1000 1170 6133 77 2 0 0 tmux
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.051132] [ 1171] 1000 1171 29914 1 2 0 0 bash
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.052362] [ 1262] 1000 1262 4022 125 2 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.053596] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.054842] [15065] 0 15065 3725 87 3 0 0 anacron
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.056115] [19925] 0 19925 4424 1 3 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.057359] [14944] 0 14944 1629 86 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.058602] [14950] 0 14950 1628 65 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.059853] [14954] 0 14954 1595 36 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.061136] [14958] 0 14958 2810 20 1 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.062414] [14961] 1000 14961 4286 459 0 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.063716] [14965] 0 14965 1618 16 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.065015] [14966] 0 14966 2290 19 2 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.066295] [14967] 0 14967 1595 37 0 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.067591] [14970] 0 14970 1628 65 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.068897] [14972] 1000 14972 4286 521 3 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.070217] [14975] 0 14975 106 8 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.071527] [14979] 1000 14979 4022 125 1 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.072832] Out of memory: Kill process 756 (NetworkManager) score 1 or sacrifice child
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4986.074110] Killed process 842 (dhclient) total-vm:86196kB, anon-rss:0kB, file-rss:4kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.242452] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.242915] systemd-cgroups cpuset=/ mems_allowed=0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.243952] Pid: 14958, comm: systemd-cgroups Not tainted 3.4.0-rc4+ #55
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.245244] Call Trace:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.246511] [<ffffffff816ae755>] ? _raw_spin_unlock+0x35/0x60
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.247766] [<ffffffff816a32a7>] dump_header+0x83/0x2eb
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.248996] [<ffffffff816ae6da>] ? _raw_spin_unlock_irqrestore+0x4a/0x90
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.250204] [<ffffffff8132e46c>] ? ___ratelimit+0xac/0x150
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.251350] [<ffffffff8115b6fc>] oom_kill_process+0x28c/0x2c0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.252505] [<ffffffff8115bc29>] out_of_memory+0x239/0x3e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.253627] [<ffffffff81161d7d>] __alloc_pages_nodemask+0xb0d/0xb20
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.254720] [<ffffffff816b2b2d>] ? sub_preempt_count+0x9d/0xd0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.255793] [<ffffffff8119fdb3>] alloc_pages_vma+0xb3/0x190
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.256780] [<ffffffff8117e248>] __do_fault+0x328/0x540
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.257795] [<ffffffff81180cf0>] handle_pte_fault+0x90/0xa10
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.258789] [<ffffffff811b7d0f>] ? mem_cgroup_count_vm_event+0x1f/0x1e0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.259769] [<ffffffff81181a18>] handle_mm_fault+0x1e8/0x2f0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.260734] [<ffffffff816b233b>] do_page_fault+0x16b/0x5d0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.261697] [<ffffffff81335a5d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.262650] [<ffffffff816af2e5>] page_fault+0x25/0x30
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.263586] Mem-Info:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.264522] Node 0 DMA per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.265473] CPU 0: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.266423] CPU 1: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.267372] CPU 2: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.268309] CPU 3: hi: 0, btch: 1 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.269234] Node 0 DMA32 per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.270149] CPU 0: hi: 186, btch: 31 usd: 2
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.271066] CPU 1: hi: 186, btch: 31 usd: 1
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.271957] CPU 2: hi: 186, btch: 31 usd: 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.272856] CPU 3: hi: 186, btch: 31 usd: 30
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.273681] Node 0 Normal per-cpu:
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.274571] CPU 0: hi: 186, btch: 31 usd: 100
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.275466] CPU 1: hi: 186, btch: 31 usd: 45
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.276351] CPU 2: hi: 186, btch: 31 usd: 87
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.277222] CPU 3: hi: 186, btch: 31 usd: 41
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278087] active_anon:1421461 inactive_anon:313478 isolated_anon:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278088] active_file:185 inactive_file:290 isolated_file:69
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278090] unevictable:3038 dirty:0 writeback:0 unstable:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278091] free:25550 slab_reclaimable:14134 slab_unreclaimable:101600
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.278092] mapped:305 shmem:86 pagetables:89707 bounce:0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.282445] Node 0 DMA free:15876kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15652kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.285312] lowmem_reserve[]: 0 3246 8034 8034
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.286303] Node 0 DMA32 free:46272kB min:27252kB low:34064kB high:40876kB active_anon:2330240kB inactive_anon:582708kB active_file:24kB inactive_file:20kB unevictable:4748kB isolated(anon):0kB isolated(file):0kB present:3324200kB mlocked:4748kB dirty:0kB writeback:0kB mapped:104kB shmem:104kB slab_reclaimable:2800kB slab_unreclaimable:132256kB kernel_stack:2296kB pagetables:133744kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:296 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.289453] lowmem_reserve[]: 0 0 4788 4788
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.290460] Node 0 Normal free:40052kB min:40196kB low:50244kB high:60292kB active_anon:3355604kB inactive_anon:671204kB active_file:716kB inactive_file:1140kB unevictable:7404kB isolated(anon):0kB isolated(file):276kB present:4902912kB mlocked:7404kB dirty:0kB writeback:0kB mapped:1116kB shmem:240kB slab_reclaimable:53736kB slab_unreclaimable:274112kB kernel_stack:4616kB pagetables:225084kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4068 all_unreclaimable? yes
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.295125] lowmem_reserve[]: 0 0 0 0
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.296327] Node 0 DMA: 1*4kB 0*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15876kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.297596] Node 0 DMA32: 179*4kB 143*8kB 158*16kB 229*32kB 136*64kB 67*128kB 9*256kB 5*512kB 2*1024kB 1*2048kB 2*4096kB = 46148kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.298883] Node 0 Normal: 449*4kB 360*8kB 263*16kB 353*32kB 150*64kB 42*128kB 10*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 39764kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.300186] 2587 total pagecache pages
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.301448] 1924 pages in swap cache
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.302721] Swap cache stats: add 1830963, delete 1829039, find 494366/494878
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.303994] Free swap = 0kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.305280] Total swap = 1023996kB
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.371694] 2097136 pages RAM
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.372170] 59776 pages reserved
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.373070] 2548580 pages shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.374252] 2007488 pages non-shared
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.375461] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.376713] [ 353] 0 353 4425 23 2 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.377924] [ 756] 0 756 67222 108 1 0 0 NetworkManager
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.379160] [ 770] 0 770 7083 1 0 0 0 systemd-logind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.380398] [ 773] 70 773 7005 50 3 0 0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.381640] [ 776] 70 776 6972 4 0 0 0 avahi-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.382883] [ 778] 0 778 1612 1 0 0 0 mcelog
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.384116] [ 783] 994 783 4893 24 0 0 0 chronyd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.385370] [ 787] 81 787 5472 69 0 -13 -900 dbus-daemon
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.386595] [ 790] 0 790 29064 41 1 0 0 ksmtuned
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.387830] [ 797] 0 797 46610 34 2 -13 -900 polkitd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.389066] [ 802] 0 802 60782 63 1 0 0 rsyslogd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.390244] [ 815] 0 815 18861 1 0 -17 -1000 sshd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.391487] [ 816] 0 816 8512 1 2 0 0 rpc.idmapd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.392739] [ 819] 0 819 4792 17 3 0 0 rpcbind
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.393980] [ 824] 29 824 6938 1 0 0 0 rpc.statd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.395227] [ 837] 0 837 27232 1 3 0 0 agetty
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.396486] [ 1170] 1000 1170 6133 112 3 0 0 tmux
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.397727] [ 1171] 1000 1171 29914 1 2 0 0 bash
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.398971] [ 1262] 1000 1262 4022 131 1 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.400219] [ 1325] 0 1325 4424 57 2 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.401493] [15065] 0 15065 3725 87 3 0 0 anacron
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.402759] [19925] 0 19925 4424 1 3 -17 -1000 udevd
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.404053] [14950] 0 14950 1628 89 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.405370] [14954] 0 14954 1595 56 3 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.406583] [14958] 0 14958 3763 32 2 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.407859] [14965] 0 14965 1621 42 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.409294] [14966] 0 14966 2810 43 3 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.410460] [14967] 0 14967 1595 59 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.411770] [14970] 0 14970 1628 89 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.413072] [14975] 0 14975 1621 55 1 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.414379] [14977] 0 14977 1750 36 2 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.415685] [14979] 1000 14979 4022 271 1 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.417022] [14980] 0 14980 1750 36 0 0 0 systemd-cgroups
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.418337] [14982] 0 14982 666 31 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.419660] [14986] 1000 14986 4319 557 2 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.420971] [14987] 1000 14987 4022 269 0 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.422278] [14988] 1000 14988 4187 402 3 0 0 trinity
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.423520] [14990] 0 14990 666 35 0 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.424793] [14992] 0 14992 104 1 2 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.426033] [14994] 0 14994 104 1 0 0 0 modprobe
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.427247] Out of memory: Kill process 756 (NetworkManager) score 1 or sacrifice child
Apr 26 13:57:15 dhcp-189-232 kernel: [ 4988.428451] Killed process 756 (NetworkManager) total-vm:268888kB, anon-rss:424kB, file-rss:8kB
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/