Re: [PATCH] Fix: disable sys_membarrier when nohz_full is enabled

[Mathieu Desnoyers]

----- On Nov 17, 2016, at 9:50 AM, rostedt rostedt@xxxxxxxxxxx wrote:

> On Thu, 17 Nov 2016 13:54:27 +0000 (UTC)
> Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx> wrote:
> 
> 
>> >> > 
>> >> > Acked-by: Lai Jiangshan <jiangshanlai@xxxxxxxxx>
>> >> > 
>> >> > But I'm afraid, in the future, tick_nohz_full will become a default y
>> >> > feature. thus it makes sys_membarrier() always disabled. we might
>> >> > need a new MEMBARRIER_CMD_XXX to handle it?
>> >> 
>> >> This may require that we send an IPI to nohz_full CPUs, which will
>> >> disturb them real-time wise. Any better ideas ?
>> > 
>> > Restrict the IPIs to CPUs running the process executing the
>> > sys_membarrier() system call.  This would mean that CPUs only
>> > are interrupted by their own application's request.
>> 
>> This would break use-cases of cross-process shared memory. :-(
> 
> Perhaps make this an opt in. That is, all processes that want to be
> affected by this can call this function with some flag that sets a flag
> in tasks struct. And have that process get an IPI even in no-hz-full
> mode if it asked to do it.

That's an interesting approach. I would be tempted to give it a
per-thread (rather than per-process) scope.

E.g., a thread could do the following to ask to be
interrupted by IPIs:

membarrier(MEMBARRIER_CMD_REGISTER_EXPEDITED, 0)

and could unregister with:

membarrier(MEMBARRIER_CMD_UNREGISTER_EXPEDITED, 0)

We can then keep a per-thread refcount internally.
(not sure the "EXPEDITED" is the right word there...
do we want it to be "NOHZ_FULL" instead ?)

Then in membarrier(MEMBARRIER_CMD_SHARED, 0), for each
nohz_full cpu, we grab the rq lock, and only send an IPI
if the running thread is registered as "expedited".

Thoughts ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com