Re: [RFC PATCH] kernel: sched: Provide a pointer to the valid CPU mask

[Thomas Gleixner]

On Thu, 6 Apr 2017, Peter Zijlstra wrote:
> On Thu, Apr 06, 2017 at 11:46:33AM +0200, Sebastian Andrzej Siewior wrote:
> > On 2017-04-06 11:32:24 [+0200], Peter Zijlstra wrote:
> > > On Tue, Apr 04, 2017 at 08:42:02PM +0200, Sebastian Andrzej Siewior wrote:
> > > > While converting the existing users I tried to stick with the rules
> > > > above howeverâ well mostly CPUFREQ tries to temporary switch the CPU
> > > > mask to do something on a certain CPU and then switches the mask back it
> > > > its original value.
> > > 
> > > 
> > > There's a bunch of that through ancient and rotten parts of the kernel.
> > > All those sites are broken.
> > > 
> > > Nothing stops userspace from setting a different affinity right after
> > > the kernel does for those threads.
> > 
> > Good. So you are saying I should convert them to something like
> > queue_work_on()?
> 
> Not sure; iirc there were a few variants. Some can indeed simply do
> queue_work_on() and possibly wait for completion. some should maybe be a
> per-cpu kthread, others will be more 'interesting'.
> 
> IIRC MIPS has a case where only 1 in N cores has an FPU. And once a task
> uses FPU, it gets affined to the core that has one or something like
> that.
>
> Of course, nothing then stops someone else breaking that affinity. But I
> suspect it will simply fault on the next FPU instruction and 'reset' the
> mask or something. I've no clue and no real desire to know.

It does nasty games with it's own storage of p->thread.user_cpus_allowed
and a fully seperate implementation of sys_sched_set|getaffinity.

Plus a magic trap handler which forces the thread to a CPU with FPU when
the user_cpus_allowed mask intersects with the cpus_with_fpu_mask...

Magic crap, which could all be replaced by a simple function in the
scheduler which allows to push a task to a FPU CPU and then disable
migration.

Thanks,

	tglx