Re: kernel BUG at net/sunrpc/svc.c:570 after updating from v5.15.153 to v5.15.155

[NeilBrown]

On Thu, 25 Apr 2024, Chuck Lever III wrote:
> 
> > On Apr 24, 2024, at 9:33 AM, Chuck Lever III <chuck.lever@xxxxxxxxxx> wrote:
> > 
> >> On Apr 24, 2024, at 3:42 AM, Chris Packham <Chris.Packham@xxxxxxxxxxxxxxxxxxx> wrote:
> >> 
> >> On 24/04/24 13:38, Chris Packham wrote:
> >>> 
> >>> On 24/04/24 12:54, Chris Packham wrote:
> >>>> Hi Jeff, Chuck, Greg,
> >>>> 
> >>>> After updating one of our builds along the 5.15.y LTS branch our 
> >>>> testing caught a new kernel bug. Output below.
> >>>> 
> >>>> I haven't dug into it yet but wondered if it rang any bells.
> >>> 
> >>> A bit more info. This is happening at "reboot" for us. Our embedded 
> >>> devices use a bit of a hacked up reboot process so that they come back 
> >>> faster in the case of a failure.
> >>> 
> >>> It doesn't happen with a proper `systemctl reboot` or with a SYSRQ+B
> >>> 
> >>> I can trigger it with `killall -9 nfsd` which I'm not sure is a 
> >>> completely legit thing to do to kernel threads but it's probably close 
> >>> to what our customized reboot does.
> >> 
> >> I've bisected between v5.15.153 and v5.15.155 and identified commit 
> >> dec6b8bcac73 ("nfsd: Simplify code around svc_exit_thread() call in 
> >> nfsd()") as the first bad commit. Based on the context that seems to 
> >> line up with my reproduction. I'm wondering if perhaps something got 
> >> missed out of the stable track? Unfortunately I'm not able to run a more 
> >> recent kernel with all of the nfs related setup that is being used on  
> >> the system in question.
> > 
> > Thanks for bisecting, that would have been my first suggestion.
> > 
> > The backport included all of the NFSD patches up to v6.2, but
> > there might be a missing server-side SunRPC patch.
> 
> So dec6b8bcac73 ("nfsd: Simplify code around svc_exit_thread()
> call in  nfsd()") is from v6.6, so it was applied to v5.15.y
> only to get a subsequent NFSD fix to apply.
> 
> The immediately previous upstream commit is missing:
> 
>   390390240145 ("nfsd: don't allow nfsd threads to be signalled.")
> 
> For testing, I've applied this to my nfsd-5.15.y branch here:
> 
>   https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git
> 
> However even if that fixes the reported crash, this suggests
> that after v6.6, nfsd threads are not going to respond to
> "killall -9 nfsd".

I think this likely is the problem.  The nfsd threads must be being
killed by a signal.
One only other cause for an nfsd thread to exit is if
svc_set_num_threads() is called, and all places that call that hold a
ref on the serv structure so the final put won't happen when the thread
exits.

Before the patch that bisect found, the nfsd thread would exit with

 svc_get();
 svc_exit_thread();
 nfsd_put();

This also holds a ref across the svc_exit_thread(), and ensures the
final 'put' happens from nfsD_put(), not svc_put() (in
svc_exit_thread()).

Chris: what was the context when the crash happened?  Could the nfsd
threads have been signalled?  That hasn't been the standard way to stop
nfsd threads for a long time, so I'm a little surprised that it is
happening.

NeilBrown