Re: [PATCH v5] locking/rwsem: Make handoff bit handling more consistent

From: john . p . donnelly
Date: Tue Apr 12 2022 - 12:29:42 EST


On 4/11/22 4:07 PM, Waiman Long wrote:

On 4/11/22 17:03, john.p.donnelly@xxxxxxxxxx wrote:


I have reached out to Waiman and he suggested this for our next test pass:


1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock path

Does this commit help to avoid the lockup problem?

Commit 1ee326196c6658 fixes a potential missed wakeup problem when a reader first in the wait queue is interrupted out without acquiring the lock. It is actually not a fix for commit d257cc8cb8d5. However, this commit changes the out_nolock path behavior of writers by leaving the handoff bit set when the wait queue isn't empty. That likely makes the missed wakeup problem easier to reproduce.

Cheers,
Longman


Hi,


We are testing now

ETA for fio soak test completion is  ~15hr from now.

I wanted to share the stack traces for future reference + occurrences.

I am looking forward to your testing results tomorrow.

Cheers,
Longman

Hi

Our 24hr fio soak test with :

1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock path


applied to 5.15.30 passed.

I suggest you append 1ee326196c6658 with :


cc: stable

Fixes: d257cc8cb8d5 ("locking/rwsem: Make handoff bit handling more consistent")


I'll leave the implementation details up to the core maintainers how to do that ;-)

...

Thank you

John.
--- Begin Message --- On 4/11/22 1:40 PM, Waiman Long wrote:
On 4/11/22 14:26, john.p.donnelly@xxxxxxxxxx wrote:
On 2/14/22 9:47 AM, chenguanyou wrote:
Hi Waiman, Greg,
This patch has been merged in branch linux-5.16.y.
Can we take it to the linux-5.10.y LTS version?

thanks,

Hi,

As a FYI:

We have observed that following lockup with this commit added to 5.15.LTS:

d257cc8cb8d5 - locking/rwsem: Make handoff bit handling more consistent (4 months ago) <Waiman Long>

The "fio" test suit fails with LVM devices composed of four NVME devices with these observed lockup, panic.



ext4:

PID: 3682   TASK: ffff8f489ae34bc0  CPU: 2   COMMAND: "dio/dm-0"
 #0 [fffffe0000083e50] crash_nmi_callback at ffffffff828772b3
 #1 [fffffe0000083e58] nmi_handle at ffffffff82840778
 #2 [fffffe0000083ea0] default_do_nmi at ffffffff8337a1e2
 #3 [fffffe0000083ec8] exc_nmi at ffffffff8337a48d
 #4 [fffffe0000083ef0] end_repeat_nmi at ffffffff8340153b
    [exception RIP: _raw_spin_lock_irq+23]
    RIP: ffffffff8338b2e7  RSP: ffff9c4409b47c78  RFLAGS: 00000046
    RAX: 0000000000000000  RBX: ffff8f489ae34bc0  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: 0000000000000000  RDI: ffff8f47f7b90104
    RBP: ffff9c4409b47d20   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000000  R12: ffff8f47f7b90104
    R13: ffff9c4409b47cb0  R14: ffff8f47f7b900f0  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    <NMI exception stack> ---
 #5 [ffff9c4409b47c78] _raw_spin_lock_irq at ffffffff8338b2e7
 #6 [ffff9c4409b47c78] rwsem_down_write_slowpath at ffffffff82925be9
 #7 [ffff9c4409b47d28] ext4_map_blocks at ffffffffc11c26dc [ext4]
 #8 [ffff9c4409b47d98] ext4_convert_unwritten_extents at
ffffffffc11ad9e0 [ext4]
 #9 [ffff9c4409b47df0] ext4_dio_write_end_io at ffffffffc11b22aa [ext4]

xfs:

PID: 3719   TASK: ffff9f81d2d74bc0  CPU: 37  COMMAND: "dio/dm-0"
 #0 [fffffe0000894e50] crash_nmi_callback at ffffffffad6772b3
 #1 [fffffe0000894e58] nmi_handle at ffffffffad640778
 #2 [fffffe0000894ea0] default_do_nmi at ffffffffae17a1e2
 #3 [fffffe0000894ec8] exc_nmi at ffffffffae17a48d
 #4 [fffffe0000894ef0] end_repeat_nmi at ffffffffae20153b
    [exception RIP: _raw_spin_lock_irq+23]
    RIP: ffffffffae18b2e7  RSP: ffffbb7ec9637c48  RFLAGS: 00000046
    RAX: 0000000000000000  RBX: ffff9f81d2d74bc0  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: 0000000000000000  RDI: ffff9f81c04a918c
    RBP: ffffbb7ec9637ce8   R8: 0000000000000000   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000000  R12: ffff9f81c04a918c
    R13: ffffbb7ec9637c80  R14: ffff9f81c04a9178  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
    <NMI exception stack> ---
 #5 [ffffbb7ec9637c48] _raw_spin_lock_irq at ffffffffae18b2e7
 #6 [ffffbb7ec9637c48] rwsem_down_write_slowpath at ffffffffad725be9
 #7 [ffffbb7ec9637cf0] xfs_trans_alloc_inode at ffffffffc074f2bd [xfs]
 #8 [ffffbb7ec9637d50] xfs_iomap_write_unwritten at ffffffffc073ad15
[xfs]
 #9 [ffffbb7ec9637dd0] xfs_dio_write_end_io at ffffffffc072db62 [xfs]


I have reached out to Waiman and he suggested this for our next test pass:


1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock path

Does this commit help to avoid the lockup problem?

Commit 1ee326196c6658 fixes a potential missed wakeup problem when a reader first in the wait queue is interrupted out without acquiring the lock. It is actually not a fix for commit d257cc8cb8d5. However, this commit changes the out_nolock path behavior of writers by leaving the handoff bit set when the wait queue isn't empty. That likely makes the missed wakeup problem easier to reproduce.

Cheers,
Longman


Hi,


We are testing now

ETA for fio soak test completion is ~15hr from now.

I wanted to share the stack traces for future reference + occurrences.


Cheers.

JD



...


--- End Message ---