Re: [PATCH] Fix: disable sys_membarrier when nohz_full is enabled

[Paul E. McKenney]

On Thu, Nov 17, 2016 at 01:54:27PM +0000, Mathieu Desnoyers wrote:
> ----- On Nov 17, 2016, at 8:40 AM, Paul E. McKenney paulmck@xxxxxxxxxxxxxxxxxx wrote:
> 
> > On Thu, Nov 17, 2016 at 11:46:34AM +0000, Mathieu Desnoyers wrote:
> >> ----- On Nov 17, 2016, at 1:51 AM, Lai Jiangshan jiangshanlai@xxxxxxxxx wrote:
> >> 
> >> > On Fri, Nov 4, 2016 at 12:29 AM, Mathieu Desnoyers
> >> > <mathieu.desnoyers@xxxxxxxxxxxx> wrote:
> >> >> Userspace applications should be allowed to expect the membarrier system
> >> >> call with MEMBARRIER_CMD_SHARED command to issue memory barriers on
> >> >> nohz_full CPUs, but synchronize_sched() does not take those into
> >> >> account.
> >> >>
> >> >> Given that we do not want unrelated processes to be able to affect
> >> >> real-time sensitive nohz_full CPUs, simply return ENOSYS when membarrier
> >> >> is invoked on a kernel with enabled nohz_full CPUs.
> >> >>
> >> >> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
> >> >> CC: "Paul E. McKenney" <paulmck@xxxxxxxxxxxxxxxxxx>
> >> >> CC: Josh Triplett <josh@xxxxxxxxxxxxxxxx>
> >> >> CC: Steven Rostedt <rostedt@xxxxxxxxxxx>
> >> >> CC: Lai Jiangshan <jiangshanlai@xxxxxxxxx>
> >> >> CC: <stable@xxxxxxxxxxxxxxx>    [3.10+]
> >> >> ---
> >> >>  kernel/membarrier.c | 4 ++++
> >> >>  1 file changed, 4 insertions(+)
> >> >>
> >> >> diff --git a/kernel/membarrier.c b/kernel/membarrier.c
> >> >> index 536c727..9f9284f 100644
> >> >> --- a/kernel/membarrier.c
> >> >> +++ b/kernel/membarrier.c
> >> >> @@ -16,6 +16,7 @@
> >> >>
> >> >>  #include <linux/syscalls.h>
> >> >>  #include <linux/membarrier.h>
> >> >> +#include <linux/tick.h>
> >> >>
> >> >>  /*
> >> >>   * Bitmask made from a "or" of all commands within enum membarrier_cmd,
> >> >> @@ -51,6 +52,9 @@
> >> >>   */
> >> >>  SYSCALL_DEFINE2(membarrier, int, cmd, int, flags)
> >> >>  {
> >> >> +       /* MEMBARRIER_CMD_SHARED is not compatible with nohz_full. */
> >> >> +       if (tick_nohz_full_enabled())
> >> >> +               return -ENOSYS;
> >> > 
> >> > I guess this code needs to be moved down into the branch of
> >> > "case MEMBARRIER_CMD_SHARED" to match its comment.
> >> 
> >> No, that would be unexpected from user-space. Either a system
> >> call is implemented or not, not "implemented for some parameters".
> >> 
> >> We also want MEMBARRIER_CMD_QUERY to return -ENOSYS in this case,
> >> and all other parameter values to also return -ENOSYS (rather than
> >> -EINVAL).
> >> 
> >> If a system call that returns successfully on CMD_QUERY or EINVAL,
> >> user-space may assume it will not have to handle ENOSYS in the
> >> next calls.
> >> 
> >> 
> >> > 
> >> > Acked-by: Lai Jiangshan <jiangshanlai@xxxxxxxxx>
> >> > 
> >> > But I'm afraid, in the future, tick_nohz_full will become a default y
> >> > feature. thus it makes sys_membarrier() always disabled. we might
> >> > need a new MEMBARRIER_CMD_XXX to handle it?
> >> 
> >> This may require that we send an IPI to nohz_full CPUs, which will
> >> disturb them real-time wise. Any better ideas ?
> > 
> > Restrict the IPIs to CPUs running the process executing the
> > sys_membarrier() system call.  This would mean that CPUs only
> > are interrupted by their own application's request.
> 
> This would break use-cases of cross-process shared memory. :-(

Good point -- getting this working does look to be good clean fun...

							Thanx, Paul