Re: [PATCH] Fix: disable sys_membarrier when nohz_full is enabled

[Mathieu Desnoyers]

----- On Nov 17, 2016, at 1:51 AM, Lai Jiangshan jiangshanlai@xxxxxxxxx wrote:

> On Fri, Nov 4, 2016 at 12:29 AM, Mathieu Desnoyers
> <mathieu.desnoyers@xxxxxxxxxxxx> wrote:
>> Userspace applications should be allowed to expect the membarrier system
>> call with MEMBARRIER_CMD_SHARED command to issue memory barriers on
>> nohz_full CPUs, but synchronize_sched() does not take those into
>> account.
>>
>> Given that we do not want unrelated processes to be able to affect
>> real-time sensitive nohz_full CPUs, simply return ENOSYS when membarrier
>> is invoked on a kernel with enabled nohz_full CPUs.
>>
>> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
>> CC: "Paul E. McKenney" <paulmck@xxxxxxxxxxxxxxxxxx>
>> CC: Josh Triplett <josh@xxxxxxxxxxxxxxxx>
>> CC: Steven Rostedt <rostedt@xxxxxxxxxxx>
>> CC: Lai Jiangshan <jiangshanlai@xxxxxxxxx>
>> CC: <stable@xxxxxxxxxxxxxxx>    [3.10+]
>> ---
>>  kernel/membarrier.c | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/kernel/membarrier.c b/kernel/membarrier.c
>> index 536c727..9f9284f 100644
>> --- a/kernel/membarrier.c
>> +++ b/kernel/membarrier.c
>> @@ -16,6 +16,7 @@
>>
>>  #include <linux/syscalls.h>
>>  #include <linux/membarrier.h>
>> +#include <linux/tick.h>
>>
>>  /*
>>   * Bitmask made from a "or" of all commands within enum membarrier_cmd,
>> @@ -51,6 +52,9 @@
>>   */
>>  SYSCALL_DEFINE2(membarrier, int, cmd, int, flags)
>>  {
>> +       /* MEMBARRIER_CMD_SHARED is not compatible with nohz_full. */
>> +       if (tick_nohz_full_enabled())
>> +               return -ENOSYS;
> 
> I guess this code needs to be moved down into the branch of
> "case MEMBARRIER_CMD_SHARED" to match its comment.

No, that would be unexpected from user-space. Either a system
call is implemented or not, not "implemented for some parameters".

We also want MEMBARRIER_CMD_QUERY to return -ENOSYS in this case,
and all other parameter values to also return -ENOSYS (rather than
-EINVAL).

If a system call that returns successfully on CMD_QUERY or EINVAL,
user-space may assume it will not have to handle ENOSYS in the
next calls.


> 
> Acked-by: Lai Jiangshan <jiangshanlai@xxxxxxxxx>
> 
> But I'm afraid, in the future, tick_nohz_full will become a default y
> feature. thus it makes sys_membarrier() always disabled. we might
> need a new MEMBARRIER_CMD_XXX to handle it?

This may require that we send an IPI to nohz_full CPUs, which will
disturb them real-time wise. Any better ideas ?

Thanks,

Mathieu

> 
> thanks,
> Lai

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com