Re: [PATCH v2 4/4] x86/static_call: Add inline static call implementation for x86-64

[Josh Poimboeuf]

On Tue, Dec 11, 2018 at 09:41:37AM +0000, David Laight wrote:
> From: Josh Poimboeuf
> > Sent: 30 November 2018 16:27
> > 
> > On Thu, Nov 29, 2018 at 03:04:20PM -0800, Linus Torvalds wrote:
> > > On Thu, Nov 29, 2018 at 12:25 PM Josh Poimboeuf <jpoimboe@xxxxxxxxxx> wrote:
> ...
> > > > Maybe that would be ok.  If my math is right, we would use the
> > > > out-of-line version almost 5% of the time due to cache misalignment of
> > > > the address.
> > >
> > > Note that I don't think cache-line alignment is necessarily sufficient.
> > >
> > > The I$ fetch from the cacheline can happen in smaller chunks, because
> > > the bus between the I$ and the instruction decode isn't a full
> > > cacheline (well, it is _now_ in modern big cores, but it hasn't always
> > > been).
> > >
> > > So even if the cacheline is updated atomically, I could imagine seeing
> > > a partial fetch from the I$ (old values) and then a second partial
> > > fetch (new values).
> > >
> > > It would be interesting to know what the exact fetch rules are.
> > 
> > I've been doing  some cross-modifying code experiments on Nehalem, with
> > one CPU writing call destinations while the other CPUs are executing
> > them.  Reliably, one of the readers goes off into the weeds within a few
> > seconds.
> > 
> > The writing was done with just text_poke(), no #BP.
> > 
> > I wasn't able to figure out the pattern in the addresses of the
> > corrupted call sites.  It wasn't cache line.
> > 
> > That was on Nehalem.  Skylake didn't crash at all.
> 
> Interesting thought?
> 
> If it is possible to add a prefix that can be overwritten by an int3
> is it also possible to add something that the assembler will use
> to align the instruction so that a write to the 4 byte offset
> will be atomic?
> 
> I'd guess that avoiding 8 byte granularity would be sufficient.
> So you'd need a 1, 2 or 3 byte nop depending on the actual
> alignment - although a 3 byte one would always do.

The problem is that the call is done in C code, and we don't have a
feasible way to use inline asm to call functions with more than five
arguments.

BTW, my original experiments (mentioned above) were a bit... flawed.  I
used text_poke(), which does memcpy(), which writes one byte at a time.
No wonder it wasn't atomic.

I'll need to do some more experiments.

-- 
Josh