Re: [PATCH] bpf: Tweak BPF jump table optimizations for objtool compatibility

[Alexei Starovoitov]

On Tue, May 05, 2020 at 03:28:23PM -0500, Josh Poimboeuf wrote:
> On Tue, May 05, 2020 at 12:53:20PM -0700, Alexei Starovoitov wrote:
> > On Tue, May 05, 2020 at 01:11:08PM -0500, Josh Poimboeuf wrote:
> > > On Tue, May 05, 2020 at 10:43:00AM -0700, Alexei Starovoitov wrote:
> > > > > Or, if you want to minimize the patch's impact on other arches, and keep
> > > > > the current patch the way it is (with bug fixed and changed patch
> > > > > description), that's fine too.  I can change the patch description
> > > > > accordingly.
> > > > > 
> > > > > Or if you want me to measure the performance impact of the +40% code
> > > > > growth, and *then* decide what to do, that's also fine.  But you'd need
> > > > > to tell me what tests to run.
> > > > 
> > > > I'd like to minimize the risk and avoid code churn,
> > > > so how about we step back and debug it first?
> > > > Which version of gcc are you using and what .config?
> > > > I've tried:
> > > > Linux version 5.7.0-rc2 (gcc version 10.0.1 20200505 (prerelease) (GCC)
> > > > CONFIG_UNWINDER_ORC=y
> > > > # CONFIG_RETPOLINE is not set
> > > > 
> > > > and objtool didn't complain.
> > > > I would like to reproduce it first before making any changes.
> > > 
> > > Revert
> > > 
> > >   3193c0836f20 ("bpf: Disable GCC -fgcse optimization for ___bpf_prog_run()")
> > > 
> > > and compile with retpolines off (and either ORC or FP, doesn't matter).
> > > 
> > > I'm using GCC 9.3.1:
> > > 
> > >   kernel/bpf/core.o: warning: objtool: ___bpf_prog_run()+0x8dc: sibling call from callable instruction with modified stack frame
> > > 
> > > That's the original issue described in that commit.
> > 
> > I see something different.
> > With gcc 8, 9, and 10 and CCONFIG_UNWINDER_FRAME_POINTER=y
> > I see:
> > kernel/bpf/core.o: warning: objtool: ___bpf_prog_run()+0x4837: call without frame pointer save/setup
> > and sure enough assembly code for ___bpf_prog_run does not countain frame setup
> > though -fno-omit-frame-pointer flag was passed at command line.
> > Then I did:
> > static u64 /*__no_fgcse*/ ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn, u64 *stack)
> > and the assembly had proper frame, but objtool wasn't happy:
> > kernel/bpf/core.o: warning: objtool: ___bpf_prog_run()+0x480a: sibling call from callable instruction with modified stack frame
> > 
> > gcc 6.3 doesn't have objtool warning with and without -fno-gcse.
> > 
> > Looks like we have two issues here.
> > First gcc 8, 9 and 10 have a severe bug with __attribute__((optimize("")))
> > In this particular case passing -fno-gcse somehow overruled -fno-omit-frame-pointer
> > which is serious issue. powerpc is using __nostackprotector. I don't understand
> > how it can keep working with newer gcc-s. May be got lucky.
> > Plenty of other projects use various __attribute__((optimize("")))
> > they all have to double check that their vesion of GCC produces correct code.
> > Can somebody reach out to gcc folks for explanation?
> 
> Right.  I've mentioned this several times now.  That's why my patch
> reverts 3193c0836f20.  I don't see any other way around it.  The GCC
> manual even says this attribute should not be used in production code.

What you mentioned in commit log is:
"It doesn't append options to the command-line arguments.  Instead
it starts from a blank slate.  And according to recent GCC documentation
it's not recommended for production use."

I don't think anyone could have guessed from such description that it kills
-fno-omit-frame-pointer but it doesn't reduce optimization level to -O0
and it doesn't kill -D, -m, -I, -std= and other flags.

As far as workaround I prefer the following: