Re: [RFC][PATCH 3/4] locking/mutex: Introduce __mutex_trylock_or_handoff()

[Peter Zijlstra]

On Wed, Jun 30, 2021 at 12:30:43PM -0400, Waiman Long wrote:
> On 6/30/21 11:35 AM, Peter Zijlstra wrote:
> > Yanfei reported that it is possible to loose HANDOFF when we race with
> > mutex_unlock() and end up setting HANDOFF on an unlocked mutex. At
> > that point anybody can steal it, loosing HANDOFF in the process.
> > 
> > If this happens often enough, we can in fact starve the top waiter.
> > 
> > Solve this by folding the 'set HANDOFF' operation into the trylock
> > operation, such that either we acquire the lock, or it gets HANDOFF
> > set. This avoids having HANDOFF set on an unlocked mutex.
> > 
> > Reported-by: Yanfei Xu <yanfei.xu@xxxxxxxxxxxxx>
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@xxxxxxxxxxxxx>
> > ---
> >   kernel/locking/mutex.c |   58 +++++++++++++++++++++++++++++--------------------
> >   1 file changed, 35 insertions(+), 23 deletions(-)
> > 
> > --- a/kernel/locking/mutex.c
> > +++ b/kernel/locking/mutex.c
> > @@ -91,10 +91,7 @@ static inline unsigned long __owner_flag
> >   	return owner & MUTEX_FLAGS;
> >   }
> > -/*
> > - * Trylock variant that returns the owning task on failure.
> > - */
> > -static inline struct task_struct *__mutex_trylock_or_owner(struct mutex *lock)
> > +static inline struct task_struct *__mutex_trylock_common(struct mutex *lock, bool handoff)
> >   {
> >   	unsigned long owner, curr = (unsigned long)current;
> > @@ -104,39 +101,56 @@ static inline struct task_struct *__mute
> >   		unsigned long task = owner & ~MUTEX_FLAGS;
> >   		if (task) {
> > -			if (likely(task != curr))
> > +			if (flags & MUTEX_FLAG_PICKUP) {
> > +				if (task != curr)
> > +					break;
> > +				flags &= ~MUTEX_FLAG_HANDOFF;
> 
> I think you mean "flags &= ~MUTEX_FLAG_PICKUP". Right:-)

Duh, yes. That's what you get trying to write patches with a kid in your
lap.. :-)