Re: [PATCH 4.14 022/284] SUNRPC: avoid race between mod_timer() and del_timer_sync()

[Trond Myklebust]

On Sun, 2022-09-11 at 12:43 +0000, David Laight wrote:
> From: NeilBrown
> > Sent: 08 September 2022 01:58
> > 
> > On Thu, 08 Sep 2022, Eugeniu Rosca wrote:
> > > Hello all,
> > > 
> > > On Mo, Apr 18, 2022 at 02:10:03 +0200, Greg Kroah-Hartman wrote:
> > > > From: NeilBrown <neilb@xxxxxxx>
> > > > 
> > > > commit 3848e96edf4788f772d83990022fa7023a233d83 upstream.
> > > > 
> > > > xprt_destory() claims XPRT_LOCKED and then calls
> > > > del_timer_sync().
> > > > Both xprt_unlock_connect() and xprt_release() call
> > > >  ->release_xprt()
> > > > which drops XPRT_LOCKED and *then*
> > > > xprt_schedule_autodisconnect()
> > > > which calls mod_timer().
> > > > 
> > > > This may result in mod_timer() being called *after*
> > > > del_timer_sync().
> > > > When this happens, the timer may fire long after the xprt has
> > > > been freed,
> > > > and run_timer_softirq() will probably crash.
> > > > 
> > > > The pairing of ->release_xprt() and
> > > > xprt_schedule_autodisconnect() is
> > > > always called under ->transport_lock.  So if we take -
> > > > >transport_lock to
> > > > call del_timer_sync(), we can be sure that mod_timer() will run
> > > > first
> > > > (if it runs at all).
> > > > 
> > > > Cc: stable@xxxxxxxxxxxxxxx
> > > > Signed-off-by: NeilBrown <neilb@xxxxxxx>
> > > > Signed-off-by: Trond Myklebust
> > > > <trond.myklebust@xxxxxxxxxxxxxxx>
> > > > Signed-off-by: Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>
> > > > ---
> > > >  net/sunrpc/xprt.c |    7 +++++++
> > > >  1 file changed, 7 insertions(+)
> > > > 
> > > > --- a/net/sunrpc/xprt.c
> > > > +++ b/net/sunrpc/xprt.c
> > > > @@ -1520,7 +1520,14 @@ static void xprt_destroy(struct rpc_xprt
> > > >          */
> > > >         wait_on_bit_lock(&xprt->state, XPRT_LOCKED,
> > > > TASK_UNINTERRUPTIBLE);
> > > > 
> > > > +       /*
> > > > +        * xprt_schedule_autodisconnect() can run after
> > > > XPRT_LOCKED
> > > > +        * is cleared.  We use ->transport_lock to ensure the
> > > > mod_timer()
> > > > +        * can only run *before* del_time_sync(), never after.
> > > > +        */
> > > > +       spin_lock(&xprt->transport_lock);
> > > >         del_timer_sync(&xprt->timer);
> > > > +       spin_unlock(&xprt->transport_lock);
> > 
> > I think it is sufficient to change the to spin_{,un}lock_bh()
> > in older kernels.  The spinlock call need to match other uses of
> > the
> > same lock.
> 
> Every time I see this patch it looks wrong.
> You need something to stop the code that is calling mod_timer()
> running after the spin_unlock().
> Now it might be that there is some other state that is
> already set - in which case you only need to wait for the
> spin_lock to be released - since it can't be obtained again
> (to start the timer).
> 
> So I'd expect to see:
>         spin_lock();
>         if (nothing_set_earlier)
>                 xprt->destroying = 1;
>         spin_unlock()
>         del_timer_sync();
> 
> Looking at the code (for a change) is looks even worse.
> 
> del_timer_sync() isn't anywhere near enough.
> All the timer callback function does is schedule some work.
> So you also need to wait for the work to complete.
> 
> Changing it all to use delayed_work might reduce the problems.
> 
> Oh, any using proper mutex/locks instead of wait_on_bit_lock().

I suggest you read the code one more time, then.

Holding the bitlock XPRT_LOCKED until the transport is completely
destroyed is what ensures that nothing will ever schedule xprt->timer
again for that transport. This patch was needed in order to fix a minor
race when xprt_unlock_connect() needs to first release XPRT_LOCKED
before scheduling the xprt->timer.

...and, no. We're not going to break our aio model by replacing
XPRT_LOCKED with a mutex. We optimise for speed of processing of the
RPC message queue, and mutexes would break that by forcing the
workqueue thread to sleep instead of just re-queuing the message until
the lock is available as we do now. The price of having to use
wait_on_bit_lock() in the final shutdown path is well worth it.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@xxxxxxxxxxxxxxx