Re: Rough notes from sys_membarrier() lightning BoF

From: Paul E. McKenney
Date: Mon Sep 18 2017 - 16:35:09 EST


On Mon, Sep 18, 2017 at 07:37:22PM +0000, Mathieu Desnoyers wrote:
> ----- On Sep 18, 2017, at 3:29 PM, Paul E. McKenney paulmck@xxxxxxxxxxxxxxxxxx wrote:
>
> > On Mon, Sep 18, 2017 at 03:04:21PM -0400, Alan Stern wrote:
> >> On Sun, 17 Sep 2017, Paul E. McKenney wrote:
> >>
> >> > Hello!
> >> >
> >> > Rough notes from our discussion last Thursday. Please reply to the
> >> > group with any needed elaborations or corrections.
> >> >
> >> > Adding Andy and Michael on CC since this most closely affects their
> >> > architectures. Also adding Dave Watson and Maged Michael because
> >> > the preferred approach requires that processes wanting to use the
> >> > lightweight sys_membarrier() do a registration step.
> >> >
> >> > Thanx, Paul
> >> >
> >> > ------------------------------------------------------------------------
> >> >
> >> > Problem:
> >> >
> >> > 1. The current sys_membarrier() introduces an smp_mb() that
> >> > is not otherwise required on powerpc.
> >> >
> >> > 2. The envisioned JIT variant of sys_membarrier() assumes that
> >> > the return-to-user instruction sequence handling any change
> >> > to the usermode instruction stream, and Andy Lutomirski's
> >> > upcoming changes invalidate this assumption. It is believed
> >> > that powerpc has a similar issue.
> >>
> >> > E. Require that threads register before using sys_membarrier() for
> >> > private or JIT usage. (The historical implementation using
> >> > synchronize_sched() would continue to -not- require registration,
> >> > both for compatibility and because there is no need to do so.)
> >> >
> >> > For x86 and powerpc, this registration would set a TIF flag
> >> > on all of the current process's threads. This flag would be
> >> > inherited by any later thread creation within that process, and
> >> > would be cleared by fork() and exec(). When this TIF flag is set,
> >>
> >> Why a TIF flag, and why clear it during fork()? If a process registers
> >> to use private expedited sys_membarrier, shouldn't that apply to
> >> threads it will create in the future just as much as to threads it has
> >> already created?
> >
> > The reason for a TIF flag is to keep this per-architecture, as only
> > powerpc and x86 need it.
> >
> > The reason for clearing it during fork() is that fork() creates a new
> > process initially having but a single thread, which might or might
> > not use sys_membarrier(). Usually not, as most instances of fork()
> > are quickly followed by exec(). In addition, if we give an error for
> > unregistered use of private sys_membarrier(), clearing on fork() gets an
> > unambiguous error instead of a silent likely failure (due to libraries
> > being confused by the fork()).
>
> I think clearing that state on fork() would be unexpected. The child process
> inherits from the parent flag in my current implementation. Clearing the
> flag is only provided through exec().
>
> Libraries don't get re-initialized on fork, only on exec(). Therefore, it
> makes sense for the child process to inherit the state from its parent.

Fair enough!

Thanx, Paul