Re: Found the commit that causes the OOMs
From: David Howells
Date: Mon Jun 29 2009 - 11:29:24 EST
Minchan Kim <minchan.kim@xxxxxxxxx> wrote:
> Totally, I can't understand this situation.
> Now, this page allocation is order zero and It is just likely GFP_HIGHUSER.
> So it's unlikely interrupt context.
>
> Buddy already has enough fallback DMA32, I think.
> Why kernel can't allocate page for order 0 ?
> Is it allocator bug ?
I don't know, but I've got you some more information.
I can reproduce the problem much, much quicker, it turns out by just running
msgctl11 from the LTP syscalls testsuite a few times. No NFSD traffic this
time to confuse the issue or any other tests.
I also managed to get a list of the most in-use slabs at the time:
002732 shmem_inode_cache 2750 800 5 1
003143 fs_cache 3180 72 53 1
003145 files_cache 3145 728 5 1
003150 mm_struct 3150 840 9 2
003152 task_xstate 3160 512 8 1
003174 sighand_cache 3180 2112 3 2
003185 task_struct 3185 1632 5 2
003192 signal_cache 3192 928 4 1
003192 task_delay_info 3304 136 28 1
003205 pid 3330 104 37 1
003262 cred_jar 3381 168 23 1
003438 size-2048 3438 2072 3 2
003589 inode_cache 3606 608 6 1
004570 size-192 4572 216 18 1
007644 size-64 7656 88 44 1
007687 sysfs_dir_cache 7733 104 37 1
007875 selinux_inode_security 7920 96 40 1
008149 dentry 8190 216 18 1
013692 size-128 13900 152 25 1
047134 vm_area_struct 47440 192 20 1
182903 size-32 183178 56 67 1
312010 avtab_node 312081 48 77 1
This is from /proc/slabinfo, with the first two columns swapped to make
sorting on it easier. I've also attached the OOM report.
David
msgctl11 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0
msgctl11 cpuset=/ mems_allowed=0
Pid: 12170, comm: msgctl11 Not tainted 2.6.31-rc1-cachefs #146
Call Trace:
[<ffffffff8107207e>] ? oom_kill_process.clone.0+0xa9/0x245
[<ffffffff81074168>] ? drain_local_pages+0x0/0x13
[<ffffffff81072345>] ? __out_of_memory+0x12b/0x142
[<ffffffff810723c6>] ? out_of_memory+0x6a/0x94
[<ffffffff81074a90>] ? __alloc_pages_nodemask+0x42e/0x51d
[<ffffffff81091546>] ? cache_alloc_refill+0x353/0x69c
[<ffffffff81031424>] ? copy_process+0x93/0x1136
[<ffffffff81091b24>] ? kmem_cache_alloc+0x83/0xc5
[<ffffffff81031424>] ? copy_process+0x93/0x1136
[<ffffffff81029da3>] ? update_curr+0x53/0xdf
[<ffffffff810820c1>] ? handle_mm_fault+0x5dd/0x62f
[<ffffffff81032606>] ? do_fork+0x13f/0x2ba
[<ffffffff81022c32>] ? do_page_fault+0x1f8/0x20d
[<ffffffff8100b0d3>] ? stub_clone+0x13/0x20
[<ffffffff8100ad6b>] ? system_call_fastpath+0x16/0x1b
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 0
CPU 1: hi: 186, btch: 31 usd: 25
Active_anon:75004 active_file:0 inactive_anon:2192
inactive_file:0 unevictable:0 dirty:0 writeback:0 unstable:0
free:2200 slab:37795 mapped:618 pagetables:60369 bounce:0
DMA free:3928kB min:60kB low:72kB high:88kB active_anon:3024kB inactive_anon:128kB active_file:0kB inactive_file:0kB unevictable:0kB present:15364kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 968 968 968
DMA32 free:4748kB min:3948kB low:4932kB high:5920kB active_anon:297092kB inactive_anon:8640kB active_file:0kB inactive_file:0kB unevictable:0kB present:992032kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 0*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3928kB
DMA32: 476*4kB 56*8kB 14*16kB 4*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 4624kB
1154 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 0kB
Total swap = 0kB
255744 pages RAM
5589 pages reserved
248214 pages shared
219940 pages non-shared
Out of memory: kill process 4164 (msgctl11) score 119366 or a child
Killed process 10211 (msgctl11)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/