Re: OOM-killer and strange RSS value in 3.9-rc7

[Han Pingtian]

On Wed, Apr 24, 2013 at 03:36:20PM +0000, Christoph Lameter wrote:
> On Wed, 24 Apr 2013, Michal Hocko wrote:
> 
> > [CCing SL.B people and linux-mm list]
> >
> > Just for quick summary. The reporter sees OOM situations with almost
> > whole memory filled with slab memory. This is a powerpc machine with 4G
> > RAM.
> 
> Boot with "slub_debug" or enable slab debugging.
> 
> > /proc/slabinfo shows that almost whole slab occupied memory is on
> 
> Please enable debugging and look at where these objects were allocated.
> 
> > It is not clear who consumes that memory and the reporter claims this is
> > vanilla 3.9-rc7 kernel without any third party modules loaded. The issue
> > seems to be present only with CONFIG_SLUB.
> 
> cat /sys/kernel/slab/kmalloc-*/alloc_calls
I have enabled "slub_debug" and here is the
/sys/kernel/slab/kmalloc-512/alloc_calls contents:

     50 .__alloc_workqueue_key+0x90/0x5d0 age=113630/116957/119419 pid=1-1730 cpus=0,6-8,13,24,26,44,53,57,60,68 nodes=1
     11 .__alloc_workqueue_key+0x16c/0x5d0 age=113814/116733/119419 pid=1-1730 cpus=0,44,68 nodes=1
     13 .add_sysfs_param.isra.2+0x80/0x210 age=115175/117994/118779 pid=1-1342 cpus=0,8,12,24,60 nodes=1
    160 .build_sched_domains+0x108/0xe30 age=119111/119120/119131 pid=1 cpus=0 nodes=1
   9000 .alloc_fair_sched_group+0xe4/0x220 age=110549/114471/117357 pid=1-2290 cpus=0-1,5,9-11,13,24,29,33,36,38,40-41,45,48-50,53,56-58,60-63,68-69,72-73,76-77,79 nodes=1
   9000 .alloc_fair_sched_group+0x114/0x220 age=110549/114471/117357 pid=1-2290 cpus=0-1,5,9-11,13,24,29,33,36,38,40-41,45,48-50,53,56-58,60-63,68-69,72-73,76-77,79 nodes=1
     79 .cgroup_mkdir+0x7c/0x5c0 age=110576/114437/117347 pid=1-1288 cpus=10-13,24,36,38,40,48,50,56-58,61,68,72,76,79 nodes=1
     21 .alloc_desc+0x50/0x120 age=117673/119166/119454 pid=0-1 cpus=0,68 nodes=1
    160 .rb_allocate_cpu_buffer+0x6c/0x320 age=119137/119261/119418 pid=1 cpus=0 nodes=1
     14 .init_syscall_trace+0x100/0x150 age=118935/118936/118939 pid=1 cpus=0 nodes=1
      2 .mempool_kmalloc+0x1c/0x30 age=118752/118752/118752 pid=1 cpus=0 nodes=1
      1 .__vmalloc_node_range+0xd8/0x2f0 age=119449 pid=0 cpus=0 nodes=1
     16 .fsnotify_alloc_group+0x40/0x130 age=113156/115688/117662 pid=1-1888 cpus=5-6,48-49,60,62-63,68,73 nodes=1
    170 .__register_sysctl_table+0xac/0x6d0 age=115164/118954/119426 pid=0-1344 cpus=0,8,12-13,60,68 nodes=1
      1 .mpi_alloc_limb_space+0x1c/0x40 age=117361 pid=1 cpus=68 nodes=1
      1 .__tty_alloc_driver+0xa0/0x210 age=118522 pid=1 cpus=68 nodes=1
      1 .__tty_alloc_driver+0xb8/0x210 age=118522 pid=1 cpus=68 nodes=1
      1 .__tty_alloc_driver+0xf0/0x210 age=118522 pid=1 cpus=68 nodes=1
      1 .pty_unix98_install+0x60/0x2e0 age=110553 pid=2259 cpus=1 nodes=1
      1 .pty_unix98_install+0x80/0x2e0 age=110553 pid=2259 cpus=1 nodes=1
      2 .vt_do_kdsk_ioctl+0x1f0/0x4d0 age=117296/117296/117296 pid=532 cpus=33 nodes=1
      1 .con_do_clear_unimap.isra.1+0xc4/0x190 age=118499 pid=1 cpus=68 nodes=1
      4 .set_inverse_transl+0xe8/0x110 age=118498/118498/118498 pid=1 cpus=68 nodes=1
      1 .hvc_alloc+0x68/0x350 age=117434 pid=1 cpus=68 nodes=1
     18 .__bus_register+0x54/0x380 age=117364/118496/119063 pid=1 cpus=0,68 nodes=1
     32 .__class_register+0x60/0x2a0 age=112982/118257/118931 pid=1-1942 cpus=0,10,16,24,68 nodes=1
      8 .loop_add+0x58/0x2d0 age=117409/117416/117422 pid=1 cpus=68 nodes=1
      1 .ops_init+0x54/0x1c0 age=116040 pid=797 cpus=38 nodes=1
      2 .alloc_netdev_mqs+0x174/0x3b0 age=115946/117343/118740 pid=1-856 cpus=0,20 nodes=1
      7 .neigh_alloc+0xb0/0x350 age=102660/112574/115225 pid=0-1756 cpus=0,2,20,45,52,79 nodes=1
      1 .qdisc_alloc+0x84/0x170 age=115227 pid=1307 cpus=20 nodes=1
      2 .inetdev_init+0x4c/0x1a0 age=115946/117221/118496 pid=1-856 cpus=20,68 nodes=1
      1 .xfrm_sysctl_init+0x4c/0xf0 age=118496 pid=1 cpus=68 nodes=1
      1 .ip6_route_net_init+0x7c/0x1c0 age=117361 pid=1 cpus=68 nodes=1
      1 .ip6_route_net_init+0xb8/0x1c0 age=117361 pid=1 cpus=68 nodes=1
      1 .ip6_route_net_init+0xf4/0x1c0 age=117361 pid=1 cpus=68 nodes=1
      1 .rtnetlink_init+0x5c/0x21c age=118932 pid=1 cpus=0 nodes=1
      3 .dm_table_create+0x54/0x110 [dm_mod] age=115780/116630/117056 pid=664-1203 cpus=7-8 nodes=1
     53 .kmem_alloc+0x9c/0x140 [xfs] age=5/98135/116999 pid=436-2439 cpus=6-7,14,18,24-27,30,53-54,56-59,65 nodes=1
      9 .rpc_new_client+0x98/0x4a0 [sunrpc] age=113458/113571/113896 pid=1671-1733 cpus=0,2-3,44 nodes=1
      4 .xprt_alloc+0x144/0x240 [sunrpc] age=113459/113677/113896 pid=1671-1708 cpus=0,44 nodes=1
      2 .xprt_alloc_slot+0x14c/0x240 [sunrpc] age=113497/113497/113498 pid=1712-1727 cpus=3,23 nodes=1
      1 .svc_prepare_thread+0xe8/0x2b0 [sunrpc] age=113886 pid=1671 cpus=70 nodes=1
      1 .svc_prepare_thread+0x108/0x2b0 [sunrpc] age=113886 pid=1671 cpus=70 nodes=1
      3 .__svc_create+0x58/0x2c0 [sunrpc] age=113508/113763/113896 pid=1671-1712 cpus=0,44,68 nodes=1
      1 .cache_create_net+0x44/0xc0 [sunrpc] age=116202 pid=797 cpus=36 nodes=1
      2 .rpc_alloc_iostats+0x20/0x40 [sunrpc] age=113896/113896/113896 pid=1671 cpus=44 nodes=1
      1 .nfsd_reply_cache_init+0xa8/0xf0 [nfsd] age=116039 pid=797 cpus=38 nodes=1
      1 .ibmveth_alloc_buffer_pool+0x28/0x130 [ibmveth] age=115228 pid=1307 cpus=20 nodes=1
      1 .nf_conntrack_pernet_init+0xa8/0x1a0 [nf_conntrack] age=115175 pid=1342 cpus=12 nodes=1
      1 .nfs4_get_state_owner+0x29c/0x4a0 [nfsv4] age=110850 pid=2272 cpus=10 nodes=1
> 
> will show where these objects are allocated.
> 
> A dump of the other fields in /sys/kernel/slab/kmalloc*/* would also be
> useful.
> 
I have dumpped all /sys/kernel/slab/kmalloc*/* in kmalloc.tar.xz and
will attach it to this mail.

Attachment: kmalloc.tar.xz
Description: application/xz