Re: perf test BPF failing on f24: fix

[Arnaldo Carvalho de Melo]

Em Wed, Aug 03, 2016 at 11:45:57PM +0900, Masami Hiramatsu escreveu:
> > If we remove vmlinux, perf should use /proc/kallsyms. I think

I am not removing vmlinux, it is being used, its just that the function
chosen by the 'perf test BPF' testcase isn't there.

So lets go again trying to chase this without missing a single step of
the way:

We start with:

[root@jouet ~]# perf test bpf
37: Test BPF filter                                          :
37.1: Test basic BPF filtering                               : FAILED!
37.2: Test BPF prologue generation                           : Skip
37.3: Test BPF relocation checker                            : Skip
[root@jouet ~]# 

Ok, so we add -v to get more information:

[root@jouet ~]# perf test -v bpf
<BIG SNIP>
bpf: config program 'func=sys_epoll_wait'
symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
bpf: config 'func=sys_epoll_wait' is ok
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_epoll_wait address found : ffffffffbd295b50
Failed to find debug information for address ffffffffbd295b50
Probe point 'sys_epoll_wait' not found.
bpf_probe: failed to convert perf probe eventsFailed to add events
selected by BPF
test child finished with -1
---- end ----
Test BPF filter subtest 0: FAILED!

--------------

See? It _is_ using /lib/modules/4.7.0+/build/vmlinux, and it should
because:

[acme@jouet linux]$ file /lib/modules/4.7.0+/build/vmlinux
/lib/modules/4.7.0+/build/vmlinux: ELF 64-bit LSB executable, x86-64,
version 1 (SYSV), statically linked,
BuildID[sha1]=a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a, not stripped
[acme@jouet linux]$

Its the kernel that is in use:

[acme@jouet linux]$ perf buildid-list --kernel
              a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a
[acme@jouet linux]$ perf buildid-list -h --kernel

 Usage: perf buildid-list [<options>]

    -k, --kernel          Show current kernel build id

[acme@jouet linux]$

And, in this vmlinux file, there is _no_ such function:

[acme@jouet linux]$ readelf -wi /lib/modules/4.7.0+/build/vmlinux | grep -w sys_epoll_wait
[acme@jouet linux]$ 

Exactly like the 'perf probe -v bpf' says:

Symbol sys_epoll_wait address found : ffffffffbd295b50
Failed to find debug information for address ffffffffbd295b50

-----

It mapped it to an address, sure, it found it in /proc/kallsyms, but
then it didn't find it in the matching vmlinux file.

Since the test was working before, when did it stop to be available on
vmlinux?

Looking at a distro kernel vmlinux file, that comes in the
kernel-debuginfo package...

[acme@jouet linux]$ readelf -wi /usr/lib/debug/usr/lib/modules/4.6.3-300.fc24.x86_64/vmlinux | grep -w sys_epoll_wait
[acme@jouet linux]$
[acme@jouet linux]$ readelf -wi /usr/lib/debug/usr/lib/modules/4.6.3-300.fc24.x86_64/vmlinux | grep -w SyS_epoll_wait
    <2cb9655>   DW_AT_name        : (indirect string, offset: 0xe0d53): SyS_epoll_wait
[acme@jouet linux]$

So the situation is the same, i.e. it seems that we were doing the fallback
from vmlinux to kallsyms and this somehow regressed, so now lets do a tools
bisect:

Ok, I tried this on a RHEL7 machine and got this:

Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0-rc3+/build/vmlinux for symbols
Open Debuginfo file: /lib/modules/4.7.0-rc3+/build/vmlinux
Try to find probe point from debuginfo.
Symbol sys_epoll_pwait address found : ffffffff8128b4a0
Matched function: SyS_epoll_pwait
Probe point found: SyS_epoll_pwait+0

So it seems that sys_epoll_wait (of sys_epoll_pwait) was never in the debuginfo:

But tooling was finding it in kallsyms, getting its address, then being able to
"reverse lookup" it to the SyS_ alias, and now this isn't happening :-\

[root@felicio ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@felicio ~]# gcc --version | head -1
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
[root@felicio ~]#
[root@felicio ~]# readelf -wi /lib/modules/4.7.0-rc3+/build/vmlinux | grep -wi sys_epoll_pwait
    <23e4444>   DW_AT_name        : (indirect string, offset: 0xe089f): SyS_epoll_pwait	
[root@felicio ~]# 

So I'll concentrate on this path of investigation...

- Arnaldo

> > the failure would gone. The problem is: when symbol searching
> > fail using vmlinux, should we fallback to kallsyms? However,
> > this is another question.
> 
> As below result shown, perf probe tries to fallback, but it seems to
> fail to find corresponding debuginfo entry...
> 
> > > [root@jouet ~]# perf probe sys_epoll_wait
> > > Failed to find debug information for address ffffffffbd295b50
> > > Probe point 'sys_epoll_wait' not found.
> > >    Error: Failed to add events.
> 
> So, at least I should investigate it.
> 
> Thanks,
> 
> > > [root@jouet ~]# perf probe SyS_epoll_wait
> > > Added new events:
> > >    probe:SyS_epoll_wait (on SyS_epoll_wait)
> > >    probe:SyS_epoll_wait_1 (on SyS_epoll_wait)
> > >    probe:SyS_epoll_wait_2 (on SyS_epoll_wait)
> > >
> > > You can now use it in all perf tools, such as:
> > >
> > > 	perf record -e probe:SyS_epoll_wait_2 -aR sleep 1
> > >
> > > [root@jouet ~]#
> > >
> > > So I am changing the BPF perf test to use the CamelCase notation alias:
> > 
> > Changing to SyS_xxx is okay, byt we still need to know the root
> > cause.
> > 
> > Thank you.
> > 
> 
> 
> -- 
> Masami Hiramatsu <mhiramat@xxxxxxxxxx>