Re: perf test BPF failing on f24: fix

[Arnaldo Carvalho de Melo]

Em Thu, Aug 04, 2016 at 03:32:21PM +0900, Masami Hiramatsu escreveu:
> On Wed, 3 Aug 2016 17:04:15 -0300
> Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:
> > Em Wed, Aug 03, 2016 at 11:45:57PM +0900, Masami Hiramatsu escreveu:
> > > > If we remove vmlinux, perf should use /proc/kallsyms. I think

> > I am not removing vmlinux, it is being used, its just that the function
> > chosen by the 'perf test BPF' testcase isn't there.

> > So lets go again trying to chase this without missing a single step of
> > the way:

> > We start with:
> > 
> > [root@jouet ~]# perf test bpf
> > 37: Test BPF filter                                          :
> > 37.1: Test basic BPF filtering                               : FAILED!
> > 37.2: Test BPF prologue generation                           : Skip
> > 37.3: Test BPF relocation checker                            : Skip
> > [root@jouet ~]# 
> > 
> > Ok, so we add -v to get more information:
> > 
> > [root@jouet ~]# perf test -v bpf
> > <BIG SNIP>
> > bpf: config program 'func=sys_epoll_wait'
> > symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> > bpf: config 'func=sys_epoll_wait' is ok
> > Looking at the vmlinux_path (8 entries long)
> > Using /lib/modules/4.7.0+/build/vmlinux for symbols
> > Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux
> > Try to find probe point from debuginfo.
> > Symbol sys_epoll_wait address found : ffffffffbd295b50
> > Failed to find debug information for address ffffffffbd295b50
> > Probe point 'sys_epoll_wait' not found.
> > bpf_probe: failed to convert perf probe eventsFailed to add events
> > selected by BPF
> > test child finished with -1
> > ---- end ----
> > Test BPF filter subtest 0: FAILED!
> > 
> > --------------
> > 
> > See? It _is_ using /lib/modules/4.7.0+/build/vmlinux, and it should
> > because:
> > 
> > [acme@jouet linux]$ file /lib/modules/4.7.0+/build/vmlinux
> > /lib/modules/4.7.0+/build/vmlinux: ELF 64-bit LSB executable, x86-64,
> > version 1 (SYSV), statically linked,
> > BuildID[sha1]=a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a, not stripped
> > [acme@jouet linux]$
> > 
> > Its the kernel that is in use:
> > 
> > [acme@jouet linux]$ perf buildid-list --kernel
> >               a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a
> 
> > [acme@jouet linux]$ perf buildid-list -h --kernel
> > 
> >  Usage: perf buildid-list [<options>]
> > 
> >     -k, --kernel          Show current kernel build id
> > 
> > [acme@jouet linux]$
> > 
> > And, in this vmlinux file, there is _no_ such function:
> > 
> > [acme@jouet linux]$ readelf -wi /lib/modules/4.7.0+/build/vmlinux | grep -w sys_epoll_wait
> > [acme@jouet linux]$ 
> > 
> > Exactly like the 'perf probe -v bpf' says:
> > 
> > Symbol sys_epoll_wait address found : ffffffffbd295b50
> > Failed to find debug information for address ffffffffbd295b50
> > 
> > -----
> > 
> > It mapped it to an address, sure, it found it in /proc/kallsyms, but
> > then it didn't find it in the matching vmlinux file.
> > 
> > Since the test was working before, when did it stop to be available on
> > vmlinux?
> 
> Would you remember the previous test was done with this kernel or other one?
> 
> Actually, I could put probes on sys_epoll_wait on F24 standard kernel.

No, you did not, you put probes in SyS_epoll_wait, because the part that
is failing for me, i.e. mapping sys_epoll_wait kallsyms's addr to
SyS_epoll_wait's addr is working for you.

I will test with the standard kernel for f24, this is my hunch as well,
that something on the 4.7.0 kernel series (or one after the
f24/ubuntu-you-use ones) is causing this, will diff the configs

more below

> -----
> [mhiramat@devbox perf]$ sudo ./perf probe -vnf sys_epoll_wait
> probe-definition(0): sys_epoll_wait 
> symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> 0 arguments
> Looking at the vmlinux_path (8 entries long)
> Using /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux for symbols
> Open Debuginfo file: /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux
> Try to find probe point from debuginfo.
> Symbol sys_epoll_wait address found : ffffffff81292d60
> Matched function: SyS_epoll_wait
> found inline addr: 0xffffffff812930f3
> Probe point found: compat_SyS_epoll_pwait+147
> found inline addr: 0xffffffff81292ed6
> Probe point found: SyS_epoll_pwait+134
> found inline addr: 0xffffffff81292d60
> Probe point found: SyS_epoll_wait+0
> Found 3 probe_trace_events.
> Opening /sys/kernel/debug/tracing//kprobe_events write=1
> Writing event: p:probe/sys_epoll_wait _text+2699507
> Writing event: p:probe/sys_epoll_wait_1 _text+2698966
> Writing event: p:probe/sys_epoll_wait_2 _text+2698592
> Added new events:
>   probe:sys_epoll_wait (on sys_epoll_wait)
>   probe:sys_epoll_wait_1 (on sys_epoll_wait)
>   probe:sys_epoll_wait_2 (on sys_epoll_wait)
> 
> You can now use it in all perf tools, such as:
> 
> 	perf record -e probe:sys_epoll_wait_2 -aR sleep 1
> 
> -----
> FYI, if perf-probe failed to find the symbol in debuginfo, it tries to find its address
> from symbol table(kallsyms, here), and then tries to convert the address to the symbol
> in debuginfo again. It seems that in your case this convert process is failed.

Yeah, that is what is failing for me.
 
> Then, could you grep DEBUG_INFO in .config? I guess your kernel enables some
> reduced debuginfo related config enabled...

If that is the case, then we better add a proper warning because this is
very subtle :-)

Checking...

[acme@jouet linux]$ grep ^CONFIG_DEBUG_ ../build/v4.7.0+/.config | grep 'INFO\|REDUCED'
CONFIG_DEBUG_INFO=y
[acme@jouet linux]$ 

Nope.

- Arnaldo