Re: [PATCH v5] locking/rwsem: Make handoff bit handling more consistent

[john . p . donnelly]

On 4/11/22 4:07 PM, Waiman Long wrote:
> On 4/11/22 17:03, john.p.donnelly@xxxxxxxxxx wrote:
> > > > I have reached out to Waiman and he suggested this for our next test 
> > > > pass:
> > > >
> > > >
> > > > 1ee326196c6658 locking/rwsem: Always try to wake waiters in 
> > > > out_nolock path
> > >
> > > Does this commit help to avoid the lockup problem?
> > >
> > > Commit 1ee326196c6658 fixes a potential missed wakeup problem when a 
> > > reader first in the wait queue is interrupted out without acquiring 
> > > the lock. It is actually not a fix for commit d257cc8cb8d5. However, 
> > > this commit changes the out_nolock path behavior of writers by 
> > > leaving the handoff bit set when the wait queue isn't empty. That 
> > > likely makes the missed wakeup problem easier to reproduce.
> > >
> > > Cheers,
> > > Longman
> >
> > Hi,
> >
> >
> > We are testing now
> >
> > ETA for fio soak test completion is  ~15hr from now.
> >
> > I wanted to share the stack traces for future reference + occurrences.
> I am looking forward to your testing results tomorrow.
>
> Cheers,
> Longman
Hi

 Our 24hr fio soak test with :

 1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock 
path


 applied to 5.15.30  passed.

 I suggest you append  1ee326196c6658 with :


 cc: stable

  Fixes: d257cc8cb8d5 ("locking/rwsem: Make handoff bit handling more 
consistent")


I'll leave the implementation details up to the core maintainers how to 
do that ;-)

...

Thank you

John.
> --- Begin Message ---
>
> On 4/11/22 1:40 PM, Waiman Long wrote:
> > On 4/11/22 14:26, john.p.donnelly@xxxxxxxxxx wrote:
> > > On 2/14/22 9:47 AM, chenguanyou wrote:
> > > > Hi Waiman, Greg,
> > > > This patch has been merged in branch linux-5.16.y.
> > > > Can we take it to the linux-5.10.y LTS version?
> > > >
> > > > thanks,
> > >
> > > Hi,
> > >
> > > As a FYI:
> > >
> > > We have observed that following lockup with this commit added to 
> > > 5.15.LTS:
> > >
> > > d257cc8cb8d5 - locking/rwsem: Make handoff bit handling more 
> > > consistent (4 months ago) <Waiman Long>
> > >
> > > The "fio" test suit fails with LVM devices composed of four NVME 
> > > devices with these observed lockup, panic.
> > >
> > >
> > >
> > > ext4:
> > >
> > > PID: 3682   TASK: ffff8f489ae34bc0  CPU: 2   COMMAND: "dio/dm-0"
> > >  #0 [fffffe0000083e50] crash_nmi_callback at ffffffff828772b3
> > >  #1 [fffffe0000083e58] nmi_handle at ffffffff82840778
> > >  #2 [fffffe0000083ea0] default_do_nmi at ffffffff8337a1e2
> > >  #3 [fffffe0000083ec8] exc_nmi at ffffffff8337a48d
> > >  #4 [fffffe0000083ef0] end_repeat_nmi at ffffffff8340153b
> > >     [exception RIP: _raw_spin_lock_irq+23]
> > >     RIP: ffffffff8338b2e7  RSP: ffff9c4409b47c78  RFLAGS: 00000046
> > >     RAX: 0000000000000000  RBX: ffff8f489ae34bc0  RCX: 0000000000000000
> > >     RDX: 0000000000000001  RSI: 0000000000000000  RDI: ffff8f47f7b90104
> > >     RBP: ffff9c4409b47d20   R8: 0000000000000000   R9: 0000000000000000
> > >     R10: 0000000000000000  R11: 0000000000000000  R12: ffff8f47f7b90104
> > >     R13: ffff9c4409b47cb0  R14: ffff8f47f7b900f0  R15: 0000000000000000
> > >     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
> > >     <NMI exception stack> ---
> > >  #5 [ffff9c4409b47c78] _raw_spin_lock_irq at ffffffff8338b2e7
> > >  #6 [ffff9c4409b47c78] rwsem_down_write_slowpath at ffffffff82925be9
> > >  #7 [ffff9c4409b47d28] ext4_map_blocks at ffffffffc11c26dc [ext4]
> > >  #8 [ffff9c4409b47d98] ext4_convert_unwritten_extents at
> > > ffffffffc11ad9e0 [ext4]
> > >  #9 [ffff9c4409b47df0] ext4_dio_write_end_io at ffffffffc11b22aa [ext4]
> > >
> > > xfs:
> > >
> > > PID: 3719   TASK: ffff9f81d2d74bc0  CPU: 37  COMMAND: "dio/dm-0"
> > >  #0 [fffffe0000894e50] crash_nmi_callback at ffffffffad6772b3
> > >  #1 [fffffe0000894e58] nmi_handle at ffffffffad640778
> > >  #2 [fffffe0000894ea0] default_do_nmi at ffffffffae17a1e2
> > >  #3 [fffffe0000894ec8] exc_nmi at ffffffffae17a48d
> > >  #4 [fffffe0000894ef0] end_repeat_nmi at ffffffffae20153b
> > >     [exception RIP: _raw_spin_lock_irq+23]
> > >     RIP: ffffffffae18b2e7  RSP: ffffbb7ec9637c48  RFLAGS: 00000046
> > >     RAX: 0000000000000000  RBX: ffff9f81d2d74bc0  RCX: 0000000000000000
> > >     RDX: 0000000000000001  RSI: 0000000000000000  RDI: ffff9f81c04a918c
> > >     RBP: ffffbb7ec9637ce8   R8: 0000000000000000   R9: 0000000000000000
> > >     R10: 0000000000000000  R11: 0000000000000000  R12: ffff9f81c04a918c
> > >     R13: ffffbb7ec9637c80  R14: ffff9f81c04a9178  R15: 0000000000000000
> > >     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
> > >     <NMI exception stack> ---
> > >  #5 [ffffbb7ec9637c48] _raw_spin_lock_irq at ffffffffae18b2e7
> > >  #6 [ffffbb7ec9637c48] rwsem_down_write_slowpath at ffffffffad725be9
> > >  #7 [ffffbb7ec9637cf0] xfs_trans_alloc_inode at ffffffffc074f2bd [xfs]
> > >  #8 [ffffbb7ec9637d50] xfs_iomap_write_unwritten at ffffffffc073ad15
> > > [xfs]
> > >  #9 [ffffbb7ec9637dd0] xfs_dio_write_end_io at ffffffffc072db62 [xfs]
> > >
> > >
> > > I have reached out to Waiman and he suggested this for our next test 
> > > pass:
> > >
> > >
> > > 1ee326196c6658 locking/rwsem: Always try to wake waiters in out_nolock 
> > > path
> >
> > Does this commit help to avoid the lockup problem?
> >
> > Commit 1ee326196c6658 fixes a potential missed wakeup problem when a 
> > reader first in the wait queue is interrupted out without acquiring the 
> > lock. It is actually not a fix for commit d257cc8cb8d5. However, this 
> > commit changes the out_nolock path behavior of writers by leaving the 
> > handoff bit set when the wait queue isn't empty. That likely makes the 
> > missed wakeup problem easier to reproduce.
> >
> > Cheers,
> > Longman
>
> Hi,
>
>
> We are testing now
>
> ETA for fio soak test completion is  ~15hr from now.
>
> I wanted to share the stack traces for future reference + occurrences.
>
>
> Cheers.
>
> JD
>
>
>
> ...
>
>
> --- End Message ---