Re: [RFC PATCH 0/2] vfs: Use per-cpu list for SB's s_inodes list

[Dave Chinner]

On Tue, Feb 16, 2016 at 08:31:18PM -0500, Waiman Long wrote:
> This patch is a replacement of my previous list batching patch -
> https://lwn.net/Articles/674105/. Compared with the previous patch,
> this one provides better performance and fairness. However, it also
> requires a bit more changes in the VFS layer.
> 
> This patchset is a derivative of Andi Kleen's patch on "Initial per
> cpu list for the per sb inode list"
> 
> https://git.kernel.org/cgit/linux/kernel/git/ak/linux-misc.git/commit/?h=hle315/combined&id=f1cf9e715a40f44086662ae3b29f123cf059cbf4
> 
> Patch 1 introduces the per-cpu list.
> 
> Patch 2 modifies the superblock and inode structures to use the per-cpu
> list. The corresponding functions that reference those structures are
> modified.
> 
> Waiman Long (2):
>   lib/percpu-list: Per-cpu list with associated per-cpu locks
>   vfs: Use per-cpu list for superblock's inode list

xfstests:xfs/013 deadlocks (running on 4GB ram disks on a 16p VM):

[135478.644495] run fstests generic/013 at 2016-02-19 10:54:51
[135479.058833] XFS (ram0): Unmounting Filesystem
[135479.149472] XFS (ram0): Mounting V5 Filesystem
[135479.154056] XFS (ram0): Ending clean mount
[135479.461571] XFS (ram0): Unmounting Filesystem
[135479.548060] XFS (ram0): Mounting V5 Filesystem
[135479.553103] XFS (ram0): Ending clean mount
[135507.633377] NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [fsstress:3390]
[135507.634572] Modules linked in:
[135507.635059] CPU: 4 PID: 3390 Comm: fsstress Not tainted 4.5.0-rc2-dgc+ #683
[135507.636108] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
[135507.637442] task: ffff88023a420000 ti: ffff88023758c000 task.ti: ffff88023758c000
[135507.638566] RIP: 0010:[<ffffffff810f4982>]  [<ffffffff810f4982>] do_raw_spin_lock+0x52/0x120
[135507.639863] RSP: 0018:ffff88023758fe10  EFLAGS: 00000246
[135507.640669] RAX: 0000000000000000 RBX: ffff880306e05ca8 RCX: 0000000000000024
[135507.641751] RDX: 0000000000000001 RSI: 00010f15431142d8 RDI: ffff880306e05ca8
[135507.642825] RBP: ffff88023758fe30 R08: 0000000000000004 R09: 0000000000000007
[135507.643897] R10: 000000000002e400 R11: 0000000000000001 R12: ffffe8fefbd01800
[135507.644966] R13: ffff880306e05db8 R14: ffff880306e05ca8 R15: ffff880306e05c20
[135507.646051] FS:  00007f3f4244f700(0000) GS:ffff88013bc80000(0000) knlGS:0000000000000000
[135507.647269] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[135507.648140] CR2: 00007f3f415cb008 CR3: 000000018c9ab000 CR4: 00000000000006e0
[135507.649232] Stack:
[135507.649559]  ffff880306e05c20 ffffe8fefbd01800 ffff880306e05db8 ffff880306e05ca8
[135507.650740]  ffff88023758fe40 ffffffff81e01d75 ffff88023758fee8 ffffffff8121727f
[135507.651924]  ffff8802b2fb76b8 ffff8802b2fb7000 0000000a00000246 ffffe8fefbd01810
[135507.653108] Call Trace:
[135507.653504]  [<ffffffff81e01d75>] _raw_spin_lock+0x15/0x20
[135507.654343]  [<ffffffff8121727f>] sync_inodes_sb+0x1af/0x280
[135507.655204]  [<ffffffff8121d8f0>] ? SyS_tee+0x3d0/0x3d0
[135507.656003]  [<ffffffff8121d905>] sync_inodes_one_sb+0x15/0x20
[135507.656891]  [<ffffffff811efede>] iterate_supers+0xae/0x100
[135507.657748]  [<ffffffff8121dc35>] sys_sync+0x35/0x90
[135507.658507]  [<ffffffff81e0232e>] entry_SYSCALL_64_fastpath+0x12/0x71
[135507.659483] Code: 00 00 48 39 43 10 0f 84 b7 00 00 00 65 8b 05 de 57 f1 7e 39 43 08 0f 84 bb 00 00 00 8b 03 85 c0 75 0d ba 01 00 00 00 f0 0f b1 13 <85> c0 74 63 4c
[135507.733374] NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [fsstress:3380]
[135507.734665] Modules linked in:
[135507.735195] CPU: 10 PID: 3380 Comm: fsstress Tainted: G             L  4.5.0-rc2-dgc+ #683
[135507.736525] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
[135507.737966] task: ffff8803814a23c0 ti: ffff880239264000 task.ti: ffff880239264000
[135507.739176] RIP: 0010:[<ffffffff817da7b9>]  [<ffffffff817da7b9>] delay_tsc+0x39/0x80
[135507.740457] RSP: 0018:ffff880239267a00  EFLAGS: 00000206
[135507.741338] RAX: 00010f23910bf4f8 RBX: ffffe8fefbd01810 RCX: 0000000000000024
[135507.742523] RDX: 00010f2300000000 RSI: 00010f23910bf4d4 RDI: 0000000000000001
[135507.743708] RBP: ffff880239267a00 R08: 000000000000000a R09: ffff880269028460
[135507.744896] R10: 00000000c383dd1a R11: 00000000001f07e0 R12: 00000000334293fd
[135507.746087] R13: 0000000000000001 R14: 0000000083214e30 R15: ffff880265b51900
[135507.747271] FS:  00007f3f4244f700(0000) GS:ffff88033bd00000(0000) knlGS:0000000000000000
[135507.748604] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[135507.749568] CR2: 00007f3f4159d008 CR3: 0000000143196000 CR4: 00000000000006e0
[135507.750753] Stack:
[135507.751111]  ffff880239267a10 ffffffff817da6bf ffff880239267a40 ffffffff810f49bc
[135507.752400]  000060fbc0001800 ffffe8fefbd01810 ffff880306e07058 0000000000000000
[135507.753697]  ffff880239267a50 ffffffff81e01d75 ffff880239267a78 ffffffff817e529a
[135507.754991] Call Trace:
[135507.755424]  [<ffffffff817da6bf>] __delay+0xf/0x20
[135507.756234]  [<ffffffff810f49bc>] do_raw_spin_lock+0x8c/0x120
[135507.757204]  [<ffffffff81e01d75>] _raw_spin_lock+0x15/0x20
[135507.758130]  [<ffffffff817e529a>] percpu_list_add+0x2a/0x70
[135507.759064]  [<ffffffff81206830>] inode_sb_list_add+0x20/0x30
[135507.760025]  [<ffffffff814f9744>] xfs_setup_inode+0x34/0x240
[135507.760972]  [<ffffffff814fb22b>] xfs_ialloc+0x36b/0x550
[135507.761874]  [<ffffffff814fb49d>] xfs_dir_ialloc+0x8d/0x260
[135507.762805]  [<ffffffff814fb92f>] xfs_create+0x25f/0x6a0
[135507.763700]  [<ffffffff814f881d>] xfs_generic_create+0xcd/0x2a0
[135507.764686]  [<ffffffff81e01dce>] ? _raw_spin_unlock+0xe/0x20
[135507.765652]  [<ffffffff814f8a26>] xfs_vn_create+0x16/0x20
[135507.766556]  [<ffffffff811f8ff2>] vfs_create+0xc2/0x120
[135507.767430]  [<ffffffff811fb6f9>] path_openat+0x1239/0x1370
[135507.768364]  [<ffffffff812b973c>] ? ext4_file_write_iter+0x21c/0x420
[135507.769430]  [<ffffffff810d2a89>] ? __might_sleep+0x49/0x80
[135507.770362]  [<ffffffff811fc8ee>] do_filp_open+0x7e/0xe0
[135507.771256]  [<ffffffff811e41f2>] ? kmem_cache_alloc+0x42/0x170
[135507.772250]  [<ffffffff81e01dce>] ? _raw_spin_unlock+0xe/0x20
[135507.773225]  [<ffffffff8120a2dc>] ? __alloc_fd+0xbc/0x170
[135507.774135]  [<ffffffff811eb2e6>] do_sys_open+0x116/0x1f0
[135507.775043]  [<ffffffff811eb41e>] SyS_creat+0x1e/0x20
[135507.775895]  [<ffffffff81e0232e>] entry_SYSCALL_64_fastpath+0x12/0x71

Cheers,

Dave.
-- 
Dave Chinner
david@xxxxxxxxxxxxx