Re: [PATCH 2/2] sched: Warn if we fail to migrate a task

[Paul Burton]

Hi Peter,

On Mon, May 28, 2018 at 05:06:56PM +0200, Peter Zijlstra wrote:
> On Sat, May 26, 2018 at 08:46:48AM -0700, Paul Burton wrote:
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 2380bc228dd0..cda3affd45b7 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -1127,7 +1127,8 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
> >  		struct migration_arg arg = { p, dest_cpu };
> >  		/* Need help from migration thread: drop lock and wait. */
> >  		task_rq_unlock(rq, p, &rf);
> > -		stop_one_cpu(cpu_of(rq), migration_cpu_stop, &arg);
> > +		ret = stop_one_cpu(cpu_of(rq), migration_cpu_stop, &arg);
> > +		WARN_ON(ret);
> >  		tlb_migrate_finish(p->mm);
> >  		return 0;
> >  	} else if (task_on_rq_queued(p)) {
> 
> I think we can trigger this at will.. Set affinity to the CPU you're
> going to take offline and offline concurrently.
> 
> It is possible for the offline to happen between task_rq_unlock() and
> stop_one_cpu(), at which point the WARM will then trigger.

Right, good point. The problem is only if stop_one_cpu() returns an
error whilst cpu_of(rq) is being brought online, not whilst it's being
offlined.

> The point is; and maybe this should be a comment somewhere; that if this
> fails, there is nothing we can do about it, and it should be fixed up by
> migrate_tasks()/select_task_rq().
> 
> There is no point in propagating the error to userspace, since if we'd
> have slightly different timing and completed the stop_one_cpu() before
> the hot-un-plug, migrate_tasks()/select_task_rq() would've had to fix up
> anyway.

I agree userspace shouldn't need to care about this but in my case
(using the test program I linked from the previous patch) this triggers
whilst the CPU is being brought online, not taken offline. That means
migrate_tasks() is not involved, and we actually just return from here
back out from a sched_setaffinity syscall & continue running the user
task on a CPU that is no longer present in the task's cpus_allowed.

I can't think of a good qualifier to limit the warning to only trigger
in that scenario though, so in reality perhaps we're best to just trust
that with patch 1 applied the problem will go away.

Thanks,
    Paul