Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling

[Peter Zijlstra]

On Thu, Nov 26, 2020 at 10:05:19AM +1100, Balbir Singh wrote:
> > @@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
> >  			 * Optimize the 'normal' case where there aren't any
> >  			 * cookies and we don't need to sync up.
> >  			 */
> > -			if (i == cpu && !need_sync && !p->core_cookie) {
> > +			if (i == cpu && !need_sync) {
> > +				if (p->core_cookie) {
> > +					/*
> > +					 * This optimization is only valid as
> > +					 * long as there are no cookies
> 
> This is not entirely true, need_sync is a function of core cookies, so I
> think this needs more clarification, it sounds like we enter this when
> the core has no cookies, but the task has a core_cookie? The term cookie
> is quite overloaded when used in the context of core vs task.

Nah, its the same. So each task gets a cookie to identify the 'group' of
tasks (possibly just itself) it is allowed to share a core with.

When we to core task selection, the core gets assigned the cookie of the
group it will run, same thing.

> Effectively from what I understand this means that p wants to be
> coscheduled, but the core itself is not coscheduling anything at the
> moment, so we need to see if we should do a sync and that sync might
> cause p to get kicked out and a higher priority class to come in?

This whole patch is about eliding code-wide task selection when it is
not required. IOW an optimization.

When there wasn't a core cookie (IOW, the previous task selection wasn't
core wide and limited) and the task we just selected for our own CPU
also didn't have a cookie (IOW it doesn't have to be core-wide) we can
skip the core wide task selection and schedule just this CPU and call it
a day.

The logic was subtly wrong, this patch fixes it. A next patch completely
rewrites it again to make it far simpler again. Don't spend time trying
to understand this patch (unless you're _that_ kind of person ;-) but
instead apply the whole thing and look at the resulting pick_next_task()
function.