Re: OOM when adding ipv6 route: How to make available moreper-cpu memory?

[Eric Dumazet]

Le vendredi 05 novembre 2010 Ã 10:19 -0700, Ben Greear a Ãcrit :
> We are testing 500 mac-vlans with IPv6 addresses on a 32-bit kernel
> (2.6.36 + ubuntu patches + our patches)  We have one
> routing table per interface.  It seems that some of them cannot
> add routes due to lack of memory.
> 
> root@lanforge-ubuntu:/home/lanforge# ip route show table 29
> root@lanforge-ubuntu:/home/lanforge# ./local/sbin/ip -6 route add default via 2001:98::1 dev eth12#15 table 29
> RTNETLINK answers: Cannot allocate memory
> 
> I see errors such as this in the logs:
> 
> [507107.846864] PERCPU: allocation failed, size=2048 align=4, failed to allocate new chunk
> [507107.846867] Pid: 3246, comm: ip Tainted: P            2.6.36-1-ct #7
> [507107.846869] Call Trace:
> [507107.846874]  [<c05d4996>] ? printk+0x2d/0x2f
> [507107.846878]  [<c021405e>] pcpu_alloc+0x32e/0x360
> [507107.846880]  [<c02140bf>] __alloc_percpu+0xf/0x20
> [507107.846883]  [<c0558d6d>] snmp_mib_init+0x3d/0x70
> [507107.846885]  [<c0585692>] ipv6_add_dev+0x132/0x350
> [507107.846887]  [<c0557f39>] ? inetdev_init+0xb9/0x180
> [507107.846889]  [<c05895ff>] addrconf_notify+0x3f/0x490
> [507107.846891]  [<c055884f>] ? inetdev_event+0x20f/0x280
> [507107.846894]  [<c05daef3>] notifier_call_chain+0x43/0x60
> [507107.846897]  [<c016df4f>] raw_notifier_call_chain+0x1f/0x30
> [507107.846899]  [<c05008ec>] call_netdevice_notifiers+0x2c/0x60
> [507107.846901]  [<c0502ecc>] register_netdevice+0x23c/0x380
> [507107.846907]  [<f80aecf7>] macvlan_common_newlink+0x187/0x2a0 [macvlan]
> [507107.846910]  [<f80ae540>] ? macvlan_setup+0x0/0x20 [macvlan]
> [507107.846912]  [<f80aee37>] macvlan_newlink+0x27/0x30 [macvlan]
> [507107.846914]  [<c0503060>] ? netif_rx+0x0/0x120
> [507107.846915]  [<c05033c0>] ? dev_forward_skb+0x0/0xf0
> [507107.846917]  [<f80aee10>] ? macvlan_newlink+0x0/0x30 [macvlan]
> [507107.846920]  [<c050ea94>] rtnl_newlink+0x464/0x590
> [507107.846921]  [<c050e7e6>] ? rtnl_newlink+0x1b6/0x590
> [507107.846927]  [<c050cd2d>] rtnetlink_rcv_msg+0x13d/0x230
> [507107.846929]  [<c05d721f>] ? _raw_spin_lock_irqsave+0x2f/0x50
> [507107.846931]  [<c050e630>] ? rtnl_newlink+0x0/0x590
> [507107.846933]  [<c050cbf0>] ? rtnetlink_rcv_msg+0x0/0x230
> [507107.846935]  [<c0522276>] netlink_rcv_skb+0x86/0xb0
> [507107.846936]  [<c050cbdc>] rtnetlink_rcv+0x1c/0x30
> [507107.846938]  [<c0521f69>] netlink_unicast+0x259/0x280
> [507107.846940]  [<c0522ba8>] netlink_sendmsg+0x1f8/0x300
> [507107.846943]  [<c04f0b79>] sock_sendmsg+0xd9/0x100
> [507107.846945]  [<c013e0dd>] ? finish_task_switch+0x3d/0xc0
> [507107.846947]  [<c04f0b79>] ? sock_sendmsg+0xd9/0x100
> [507107.846950]  [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0
> [507107.846957]  [<f88bd559>] ? h_d_revalidate+0xe9/0x240 [aufs]
> [507107.846958]  [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0
> [507107.846962]  [<c0364c7d>] ? _copy_from_user+0x3d/0x130
> [507107.846965]  [<c04f947a>] ? verify_iovec+0x5a/0xa0
> [507107.846966]  [<c04f112d>] sys_sendmsg+0x15d/0x290
> [507107.846972]  [<f88c05f1>] ? aufs_fault+0xf1/0x110 [aufs]
> [507107.846974]  [<c013361d>] ? kmap_atomic_prot+0xcd/0xf0
> [507107.846977]  [<c01f8726>] ? handle_mm_fault+0x146/0x400
> [507107.846979]  [<c05dac0d>] ? do_page_fault+0x1cd/0x470
> [507107.846981]  [<c023419d>] ? alloc_fd+0xbd/0xf0
> [507107.846983]  [<c0364c7d>] ? _copy_from_user+0x3d/0x130
> [507107.846986]  [<c04f183b>] sys_socketcall+0xeb/0x2a0
> [507107.846989]  [<c0151d99>] ? irq_exit+0x39/0x70
> [507107.846991]  [<c021bb2e>] ? sys_open+0x2e/0x40
> [507107.846993]  [<c05d77a4>] syscall_call+0x7/0xb
> 
> 
> But, it appears to me that we have plenty of memory available.
> 
> top - 10:09:13 up 6 days, 15:56,  9 users,  load average: 0.37, 0.39, 0.45
> Tasks: 211 total,   2 running, 209 sleeping,   0 stopped,   0 zombie
> Cpu(s):  0.7%us,  2.1%sy,  0.0%ni, 97.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
> Mem:   3604332k total,  1510252k used,  2094080k free,    84928k buffers
> Swap:   154620k total,        0k used,   154620k free,  1045912k cached
> 
>    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 
> 
>       root@lanforge-ubuntu:/home/lanforge# cat /proc/meminfo
> MemTotal:        3604332 kB
> MemFree:         2094568 kB
> Buffers:           84928 kB
> Cached:          1045920 kB
> SwapCached:            0 kB
> Active:           466784 kB
> Inactive:         805632 kB
> Active(anon):     148988 kB
> Inactive(anon):     6788 kB
> Active(file):     317796 kB
> Inactive(file):   798844 kB
> Unevictable:           0 kB
> Mlocked:               0 kB
> HighTotal:       2752008 kB
> HighFree:        1512380 kB
> LowTotal:         852324 kB
> LowFree:          582188 kB
> SwapTotal:        154620 kB
> SwapFree:         154620 kB
> Dirty:               492 kB
> Writeback:             0 kB
> AnonPages:        141316 kB
> Mapped:            39428 kB
> Shmem:             14208 kB
> Slab:             170788 kB
> SReclaimable:      81904 kB
> SUnreclaim:        88884 kB
> KernelStack:        2520 kB
> PageTables:         4516 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:     1956784 kB
> Committed_AS:     582604 kB
> VmallocTotal:     122880 kB
> VmallocUsed:       48592 kB
> VmallocChunk:      24280 kB
> HardwareCorrupted:     0 kB
> HugePages_Total:       0
> HugePages_Free:        0
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       4096 kB
> DirectMap4k:       12280 kB
> DirectMap4M:      897024 kB
> 
> 
> Is there any way to tune the system so that it has more memory available
> for per-cpu data structures?

CC linux-kernel and Tejun Heo

How many possible cpus do you have ?
head -1 /proc/interrupts

and please post :
cat /proc/vmallocinfo

Thanks


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/