Re: [PATCH 7/8] membarrier: Remove arm (32) support for SYNC_CORE

[Mark Rutland]

On Thu, Jun 17, 2021 at 07:00:26AM -0700, Andy Lutomirski wrote:
> 
> 
> On Thu, Jun 17, 2021, at 6:51 AM, Mark Rutland wrote:
> > On Thu, Jun 17, 2021 at 06:41:41AM -0700, Andy Lutomirski wrote:
> 
> > > In any event, I’m even more convinced that no new SYNC_CORE arches
> > > should be added. We need a new API that just does the right thing. 
> > 
> > My intuition is the other way around, and that this is a gnereally
> > useful thing for architectures that require context synchronization.
> 
> Except that you can't use it in a generic way.  You have to know the
> specific rules for your arch.

That's generally true for modifying instruction streams though? The man
page for cacheflush(2) calls out that it is not portable.

I think what's necessary here is some mandatory per-arch documentation?

> > It's not clear to me what "the right thing" would mean specifically, and
> > on architectures with userspace cache maintenance JITs can usually do
> > the most optimal maintenance, and only need help for the context
> > synchronization.
> > 
> 
> This I simply don't believe -- I doubt that any sane architecture
> really works like this.  I wrote an email about it to Intel that
> apparently generated internal discussion but no results.  Consider:
> 
> mmap(some shared library, some previously unmapped address);
> 
> this does no heavyweight synchronization, at least on x86.  There is
> no "serializing" instruction in the fast path, and it *works* despite
> anything the SDM may or may not say.

Sure, and I called this case out specifically when I said:

|   * Where we can guarantee that a CPU cannot possibly have an
|     instruction in-flight (e.g. due to a lack of a mapping to fetch
|     instructions from), nothing is necessary. This is what we rely on
|     when faulting in code pages. In these cases, the CPU is liable to
|     take fault on the missing translation anyway.

.. what really matters is whether the CPU had the oppoprtunity to fetch
something stale; the context synchronization is necessary to discard
that.

Bear in mind that in many cases where this could occur in theory, we
don't hit in practice because CPUs don't happen to predict/speculate as
aggressively as they are permitted to. On arm/arm64 it's obvious that
this is a problem because the documentation clearly defines the
boundaries of what a CPU is permitted to do, whereas on other
architectures docuentation is not necessarily as clear whether this is
permited or whether the architecture mandates additional guarantees.

> We can and, IMO, should develop a sane way for user programs to
> install instructions into VMAs, for security-conscious software to
> verify them (by splitting the read and write sides?), and for their
> consumers to execute them, without knowing any arch details.  And I
> think this can be done with no IPIs except for possible TLB flushing
> when needed, at least on most architectures.  It would require a
> nontrivial amount of design work, and it would not resemble
> sys_cacheflush() or SYNC_CORE.

I'm not opposed to adding new interfaces for stuff like that, but I
don't think that membarrier(SYNC_CORE) or cacheflush(2) are necessarily
wrong as-is.

Thanks,
Mark.