Re: Change eats memory on my server
From: Eli Cohen
Date: Mon Jan 18 2021 - 04:25:35 EST
On Mon, Jan 18, 2021 at 08:57:26AM +0100, Christian König wrote:
> Am 18.01.21 um 08:49 schrieb Eli Cohen:
> > On Mon, Jan 18, 2021 at 08:43:12AM +0100, Christian König wrote:
> > > Hi Eli,
> > >
> > > have you already tried using kmemleak?
> > >
> > > This sounds like a leak of memory allocated using kmalloc(), so kmemleak
> > > should be able to catch it.
> > >
> > Hi Christian,
> >
> > I have the following configured but I did not see any visible complaint
> > in dmesg.
> >
> > CONFIG_HAVE_DEBUG_KMEMLEAK=y
> > CONFIG_DEBUG_KMEMLEAK=y
> > CONFIG_DEBUG_KMEMLEAK_MEM_POOL_SIZE=16000
> >
> > Any other configuration that I need to set?
>
> As long as you don't have any kernel parameters to enable it I think you
> need to do "echo scan > /sys/kernel/debug/kmemleak" to start a scan.
>
> The result can then be queried using "cat /sys/kernel/debug/kmemleak".
>
There are some minor leaks that I noticed a while ago coming from SE
Linux. I don't think these leaks are killing my server but here they
are. Maybe someone from SELInux would like to address them.
unreferenced object 0xffff8884a5cd5000 (size 512):
comm "swapper/0", pid 1, jiffies 4294736382 (age 8097.039s)
hex dump (first 32 bytes):
03 00 00 00 05 00 00 00 03 00 00 00 00 00 00 00 ................
00 00 00 00 00 00 00 00 00 00 00 00 ad 4e ad de .............N..
backtrace:
[<0000000028e4d3ae>] selinux_sb_alloc_security+0x2e/0xf0
[<000000009037afcc>] security_sb_alloc+0x2b/0x50
[<00000000a8f69eea>] alloc_super+0x140/0x590
[<00000000b417f227>] sget_fc+0xa9/0x380
[<0000000041b639cf>] get_tree_single+0x26/0x100
[<00000000bf572b76>] vfs_get_tree+0x4c/0x140
[<00000000c0aa3dd6>] vfs_kern_mount.part.0+0x75/0xd0
[<00000000aa61ad1d>] kern_mount+0x2f/0x60
[<000000006ce5ffac>] init_sel_fs+0xf6/0x1a6
[<00000000d3ba532d>] do_one_initcall+0xbb/0x3a0
[<0000000084b518fb>] do_initcalls+0xff/0x129
[<00000000a0cc02b2>] kernel_init_freeable+0x14c/0x178
[<000000005767353a>] kernel_init+0xd/0x120
[<00000000d425dea7>] ret_from_fork+0x22/0x30
unreferenced object 0xffff888498ae6a78 (size 8):
comm "(journald)", pid 379, jiffies 4294738985 (age 8094.447s)
hex dump (first 8 bytes):
01 00 00 00 00 00 00 00 ........
backtrace:
[<00000000727257f3>] selinux_key_alloc+0x33/0xa0
[<0000000023fcc23d>] security_key_alloc+0x3b/0x60
[<000000009b8f5c5c>] key_alloc+0x46e/0x900
[<00000000a49c5ee1>] keyring_alloc+0x27/0x70
[<0000000047d4e2e0>] install_session_keyring_to_cred+0xd7/0x120
[<0000000092fa69fa>] join_session_keyring+0x109/0x1b0
[<00000000c31be2c8>] __do_sys_keyctl+0x2c2/0x310
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff888498ae6aa0 (size 8):
comm "(journald)", pid 379, jiffies 4294738985 (age 8094.447s)
hex dump (first 8 bytes):
01 00 00 00 00 00 00 00 ........
backtrace:
[<00000000727257f3>] selinux_key_alloc+0x33/0xa0
[<0000000023fcc23d>] security_key_alloc+0x3b/0x60
[<000000009b8f5c5c>] key_alloc+0x46e/0x900
[<00000000d752137d>] key_create_or_update+0x45a/0x760
[<00000000cfad8dc7>] __do_sys_add_key+0x144/0x2a0
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff88812bdab6e0 (size 8):
comm "(lymouthd)", pid 573, jiffies 4294744193 (age 8089.252s)
hex dump (first 8 bytes):
01 00 00 00 00 00 00 00 ........
backtrace:
[<00000000727257f3>] selinux_key_alloc+0x33/0xa0
[<0000000023fcc23d>] security_key_alloc+0x3b/0x60
[<000000009b8f5c5c>] key_alloc+0x46e/0x900
[<00000000a49c5ee1>] keyring_alloc+0x27/0x70
[<0000000047d4e2e0>] install_session_keyring_to_cred+0xd7/0x120
[<0000000092fa69fa>] join_session_keyring+0x109/0x1b0
[<00000000c31be2c8>] __do_sys_keyctl+0x2c2/0x310
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff88812bdab708 (size 8):
comm "(lymouthd)", pid 573, jiffies 4294744193 (age 8089.252s)
hex dump (first 8 bytes):
01 00 00 00 00 00 00 00 ........
backtrace:
[<00000000727257f3>] selinux_key_alloc+0x33/0xa0
[<0000000023fcc23d>] security_key_alloc+0x3b/0x60
[<000000009b8f5c5c>] key_alloc+0x46e/0x900
[<00000000d752137d>] key_create_or_update+0x45a/0x760
[<00000000cfad8dc7>] __do_sys_add_key+0x144/0x2a0
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff8884945e6c00 (size 32):
comm "plymouthd", pid 585, jiffies 4294744220 (age 8089.268s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01 00 00 00 03 00 00 00 18 00 00 00 00 00 00 00 ................
backtrace:
[<0000000036de713e>] selinux_sk_alloc_security+0x55/0xc0
[<00000000fb9da50f>] security_sk_alloc+0x39/0x60
[<000000007dc46eaf>] sk_prot_alloc+0x71/0x190
[<00000000ed8b4081>] sk_alloc+0x2c/0x3c0
[<000000007ae589bb>] unix_create1+0x94/0x310
[<000000002da4459b>] unix_create+0x6b/0xd0
[<0000000022417d2a>] __sock_create+0x203/0x3c0
[<0000000099e43719>] __sys_socket+0xb6/0x160
[<00000000a9850552>] __x64_sys_socket+0x3d/0x50
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff888363795940 (size 32):
comm "plymouthd", pid 585, jiffies 4294744222 (age 8089.267s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01 00 00 00 03 00 00 00 2b 00 00 00 00 00 00 00 ........+.......
backtrace:
[<0000000036de713e>] selinux_sk_alloc_security+0x55/0xc0
[<00000000fb9da50f>] security_sk_alloc+0x39/0x60
[<00000000fd684046>] sk_prot_alloc+0x144/0x190
[<00000000ed8b4081>] sk_alloc+0x2c/0x3c0
[<000000009df0b02a>] __netlink_create+0x4b/0x180
[<0000000046434ad4>] netlink_create+0x186/0x330
[<0000000022417d2a>] __sock_create+0x203/0x3c0
[<0000000099e43719>] __sys_socket+0xb6/0x160
[<00000000a9850552>] __x64_sys_socket+0x3d/0x50
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff88811baa5ac0 (size 32):
comm "(ystemctl)", pid 677, jiffies 4294749096 (age 8084.410s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01 00 00 00 01 00 00 00 18 00 00 00 00 00 00 00 ................
backtrace:
[<0000000036de713e>] selinux_sk_alloc_security+0x55/0xc0
[<00000000fb9da50f>] security_sk_alloc+0x39/0x60
[<000000007dc46eaf>] sk_prot_alloc+0x71/0x190
[<00000000ed8b4081>] sk_alloc+0x2c/0x3c0
[<000000007ae589bb>] unix_create1+0x94/0x310
[<000000002da4459b>] unix_create+0x6b/0xd0
[<0000000022417d2a>] __sock_create+0x203/0x3c0
[<0000000099e43719>] __sys_socket+0xb6/0x160
[<00000000a9850552>] __x64_sys_socket+0x3d/0x50
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff88811baa5a80 (size 32):
comm "(ystemctl)", pid 677, jiffies 4294749096 (age 8084.410s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01 00 00 00 01 00 00 00 18 00 00 00 00 00 00 00 ................
backtrace:
[<0000000036de713e>] selinux_sk_alloc_security+0x55/0xc0
[<00000000fb9da50f>] security_sk_alloc+0x39/0x60
[<000000007dc46eaf>] sk_prot_alloc+0x71/0x190
[<00000000ed8b4081>] sk_alloc+0x2c/0x3c0
[<000000007ae589bb>] unix_create1+0x94/0x310
[<00000000bf10ff2c>] unix_stream_connect+0x154/0x990
[<00000000a535f0b9>] __sys_connect+0xfd/0x130
[<000000006ffa2364>] __x64_sys_connect+0x3e/0x50
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff88810c60d6b8 (size 8):
comm "(ystemctl)", pid 677, jiffies 4294749097 (age 8084.455s)
hex dump (first 8 bytes):
01 00 00 00 00 00 00 00 ........
backtrace:
[<00000000727257f3>] selinux_key_alloc+0x33/0xa0
[<0000000023fcc23d>] security_key_alloc+0x3b/0x60
[<000000009b8f5c5c>] key_alloc+0x46e/0x900
[<00000000a49c5ee1>] keyring_alloc+0x27/0x70
[<0000000047d4e2e0>] install_session_keyring_to_cred+0xd7/0x120
[<0000000092fa69fa>] join_session_keyring+0x109/0x1b0
[<00000000c31be2c8>] __do_sys_keyctl+0x2c2/0x310
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff88810c60d6e0 (size 8):
comm "(ystemctl)", pid 677, jiffies 4294749097 (age 8084.455s)
hex dump (first 8 bytes):
01 00 00 00 00 00 00 00 ........
backtrace:
[<00000000727257f3>] selinux_key_alloc+0x33/0xa0
[<0000000023fcc23d>] security_key_alloc+0x3b/0x60
[<000000009b8f5c5c>] key_alloc+0x46e/0x900
[<00000000d752137d>] key_create_or_update+0x45a/0x760
[<00000000cfad8dc7>] __do_sys_add_key+0x144/0x2a0
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff8884c10ab380 (size 32):
comm "systemctl", pid 677, jiffies 4294749145 (age 8084.407s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01 00 00 00 01 00 00 00 18 00 00 00 00 00 00 00 ................
backtrace:
[<0000000036de713e>] selinux_sk_alloc_security+0x55/0xc0
[<00000000fb9da50f>] security_sk_alloc+0x39/0x60
[<000000007dc46eaf>] sk_prot_alloc+0x71/0x190
[<00000000ed8b4081>] sk_alloc+0x2c/0x3c0
[<000000007ae589bb>] unix_create1+0x94/0x310
[<000000002da4459b>] unix_create+0x6b/0xd0
[<0000000022417d2a>] __sock_create+0x203/0x3c0
[<0000000099e43719>] __sys_socket+0xb6/0x160
[<00000000a9850552>] __x64_sys_socket+0x3d/0x50
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
unreferenced object 0xffff8884c10ab2c0 (size 32):
comm "systemctl", pid 677, jiffies 4294749145 (age 8084.407s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01 00 00 00 01 00 00 00 18 00 00 00 00 00 00 00 ................
backtrace:
[<0000000036de713e>] selinux_sk_alloc_security+0x55/0xc0
[<00000000fb9da50f>] security_sk_alloc+0x39/0x60
[<000000007dc46eaf>] sk_prot_alloc+0x71/0x190
[<00000000ed8b4081>] sk_alloc+0x2c/0x3c0
[<000000007ae589bb>] unix_create1+0x94/0x310
[<00000000bf10ff2c>] unix_stream_connect+0x154/0x990
[<00000000a535f0b9>] __sys_connect+0xfd/0x130
[<000000006ffa2364>] __x64_sys_connect+0x3e/0x50
[<00000000a99bb85a>] do_syscall_64+0x33/0x40
[<00000000acf36f32>] entry_SYSCALL_64_after_hwframe+0x44/0xa9