Re: [perf] perf probe fails sometimes on 4.9

From: Masami Hiramatsu
Date: Tue Jan 30 2018 - 00:44:28 EST

Next message: Joel Stanley: "[PATCH 0/2] fsi: add property to avoid scanning at boot"
Previous message: zhangheng (AC): "RE: [PATCH RFC 01/16] prcu: Add PRCU implementation"
In reply to: Pintu Kumar: "Re: [perf] perf probe fails sometimes on 4.9"
Next in thread: Pintu Kumar: "Re: [perf] perf probe fails sometimes on 4.9"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Mon, 29 Jan 2018 22:00:52 +0530
Pintu Kumar <pintu.ping@xxxxxxxxx> wrote:

> Dear Masami,
>
> Thank you so much for your reply.
> Please find some of my answers inline.
>
>
> On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote:
> > On Mon, 29 Jan 2018 13:40:34 +0530
> > Pintu Kumar <pintu.ping@xxxxxxxxx> wrote:
> >
> >> Hi All,
> >>
> >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
> >> # perf probe --add schedule
> >> schedule is out of .text, skip it.
> >> Error: Failed to add events.
> >>
> >> If any one have come across this problem please let me know the cause.
> >
> > Hi Pintu,
> >
> > Could you run it with --vv?
> >
> Ok, I will send verbose output by tomorrow.
>
> >>
> >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
> >
> > Without it, you can not probe source-level probe nor trace local variable.
> >
>
> Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
> However, I will try to manually include "-g" option during compilation.
>
> >> However, I manually copied the vmlinux file to /boot/ directory, but
> >> still it does not work.
> >
> > That doesn't work.
> > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> > Without that option, debuginfo is not generated with vmlinux.
> >
> >>
> >> I checked upstream patches until 4.15 but could not find any clue.
> >> Please let me know if there is any fixes available for this.
> >
> > Could you also ensure that you run perf by root user?
> >
>
> Yes I am running with root user.
>
> My concern is sometimes it works but sometimes it fails.

What I thought was that your kernel enables kptr_strict(but maybe not.)
Can you also try to find "schedule" and "_etext" functions in
/proc/kallsyms?

> I still needs to figure out, in which condition it works and which
> condition it fails.

Yeah, it is important.

> Usually I noticed that in fresh reboot case it works.

So, after a while, it doesn't work again? If so, it sounds like a daemon
process changes settings in background.

Thank you,

>
>
> > Thank you,
> >
> >
> >>
> >>
> >> Thank You!
> >> Regards,
> >> Pintu
> >>
> >>
> >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@xxxxxxxxx> wrote:
> >> > Hi,
> >> >
> >> > ** Changed the subject now, since these issues are related to general
> >> > perf commands.
> >> >
> >> > Following are the issues:
> >> >
> >> > 1) perf probe --add schedule - FAILED
> >> > output:
> >> > schedule is out of .text, skip it.
> >> > Error: Failed to add events.
> >> >
> >> > what is the issue here?
> >> > Sometimes it pass and sometimes it fails...
> >> > Similar is the case of 'perf inject' as well.
> >> >
> >> > 2) perf test - 1 FAILURE
> >> > 37.1: Test basic BPF filtering : FAILED!
> >> > 37.2: Test BPF prologue generation : Skip
> >> > 37.3: Test BPF relocation checker : Skip
> >> >
> >> > bpf: config program 'func=SyS_epoll_wait'
> >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
> >> > bpf: config 'func=SyS_epoll_wait' is ok
> >> > Looking at the vmlinux_path (8 entries long)
> >> > Using /boot/vmlinux for symbols
> >> > Could not open debuginfo. Try to use symbols.
> >> > SyS_epoll_wait is out of .text, skip it.
> >> > bpf_probe: failed to convert perf probe eventsFailed to add events
> >> > selected by BPF
> >> > test child finished with -1
> >> > ---- end ----
> >> > Test BPF filter subtest 0: FAILED!
> >> >
> >> > Looks like both 1,2 are related.
> >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
> >> > to /boot/ folder.
> >> >
> >> > -------
> >> > Some more info:
> >> >
> >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
> >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >> > unset env: KBUILD_OPTS
> >> > include option is set to -nostdinc -isystem
> >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> >> > set env: NR_CPUS=8
> >> > set env: LINUX_VERSION_CODE=0x40914
> >> > set env: CLANG_EXEC=/usr/bin/clang
> >> > set env: CLANG_OPTIONS=-xc
> >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
> >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
> >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
> >> > -I./include -I./arch/x86/include/uapi -I./include/uapi
> >> > -I./include/generated/uapi -include ./include/linux/kconfig.h
> >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
> >> >
> >> >
> >> > If you have any clue about these failure please help me.
> >> >
> >> >
> >> > Thanks,
> >> > Pintu
> >> >
> >> >
> >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@xxxxxxxxx> wrote:
> >> >> Hi,
> >> >>
> >> >> Thanks for your help.
> >> >> Yes it was a sub version issue.
> >> >>
> >> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
> >> >>
> >> >> make -j8 bindeb-pkg SUBLEVEL=20
> >> >>
> >> >> After passing the correct sublevel now the test is working.
> >> >>
> >> >> But still following are failing:
> >> >>
> >> >> 16: Try 'import perf' in python, checking link problems : FAILED!
> >> >> 37.2: Test BPF prologue generation : FAILED!
> >> >>
> >> >>
> >> >> This is the error I get:
> >> >>
> >> >> 16: Try 'import perf' in python, checking link problems :
> >> >> --- start ---
> >> >> test child forked, pid 7637
> >> >> Traceback (most recent call last):
> >> >> File "<stdin>", line 1, in <module>
> >> >> ImportError: No module named perf
> >> >> test child finished with -1
> >> >> ---- end ----
> >> >> Try 'import perf' in python, checking link problems: FAILED!
> >> >>
> >> >>
> >> >> Looking at the vmlinux_path (8 entries long)
> >> >> symsrc__init: cannot get elf header.
> >> >> Failed to find the path for kernel: Invalid ELF file
> >> >> bpf_probe: failed to convert perf probe eventsFailed to add events
> >> >> selected by BPF
> >> >> test child finished with -1
> >> >> ---- end ----
> >> >> Test BPF filter subtest 1: FAILED!
> >> >>
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Pintu
> >> >>
> >> >>
> >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@xxxxxxxxxx> wrote:
> >> >>>
> >> >>>
> >> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
> >> >>>>
> >> >>>> Hi All,
> >> >>>>
> >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
> >> >>>> kernel version 4.9.20.
> >> >>>> I have installed several others packages including: clang, llvm
> >> >>>>
> >> >>>> But, when I run 'perf test' I get some FAILURE.
> >> >>>> Specially, 'perf test LLVM' is failing.
> >> >>>>
> >> >>>> Please check the below error logs:
> >> >>>>
> >> >>>> ----------------------------------------------------
> >> >>>> # perf test LLVM
> >> >>>> 35: Test LLVM searching and compiling :
> >> >>>> 35.1: Basic BPF llvm compiling test : FAILED!
> >> >>>> 35.2: Test kbuild searching : Skip
> >> >>>> 35.3: Compile source for BPF prologue generation test : Skip
> >> >>>> 35.4: Compile source for BPF relocation test : Skip
> >> >>>> ----------------------------------------------------
> >> >>>>
> >> >>>> When I run with -v I get this:
> >> >>>>
> >> >>>> -----------------------------------------------------
> >> >>>> # perf test -v LLVM
> >> >>>> 35: Test LLVM searching and compiling :
> >> >>>> 35.1: Basic BPF llvm compiling test :
> >> >>>> --- start ---
> >> >>>> test child forked, pid 3304
> >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >> >>>> WARNING: unable to get correct kernel building directory.
> >> >>>> Hint: Set correct kbuild directory using 'kbuild-dir' option in [llvm]
> >> >>>> section of ~/.perfconfig or set it to "" to suppress kbuild
> >> >>>> detection.
> >> >>>>
> >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
> >> >>>>
> >> >>>>
> >> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
> >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
> >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
> >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
> >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
> >> >>>> -
> >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
> >> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
> >> >>>> test child finished with -1
> >> >>>> ---- end ----
> >> >>>> Test LLVM searching and compiling subtest 0: FAILED!
> >> >>>> -----------------------------------------------------
> >> >>>>
> >> >>>>
> >> >>>> Basic BPF LLVM compiling test is failing.
> >> >>>> It reports that bpf could not able to read the kernel version from uname.
> >> >>>> Is it because of this that 'perf test LLVM' is failing ?
> >> >>>>
> >> >>>> My uname says:
> >> >>>> 4.9-xxxx-amd-x86-64
> >> >>>
> >> >>>
> >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
> >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
> >> >>> report
> >> >>> kernel version number (from your uname string, sublevel can't be
> >> >>> determined),
> >> >>> please help us improve that function like what we have done for ubuntu
> >> >>> (checking /proc/version_signature).
> >> >>>
> >> >>>
> >> >>>
> >> >>>> My kernel version is: 4.9.20 (from Makefile)
> >> >>>>
> >> >>>> So, I dont think there is any problem with kernel version.
> >> >>>>
> >> >>>> If any body have come across this problem please let me know how to
> >> >>>> resolve this issue.
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> Thank You!
> >> >>>>
> >> >>>> Regards,
> >> >>>> Pintu
> >> >>>
> >> >>>
> >> >>>
> >
> >
> > --
> > Masami Hiramatsu <mhiramat@xxxxxxxxxx>

--
Masami Hiramatsu <mhiramat@xxxxxxxxxx>

Next message: Joel Stanley: "[PATCH 0/2] fsi: add property to avoid scanning at boot"
Previous message: zhangheng (AC): "RE: [PATCH RFC 01/16] prcu: Add PRCU implementation"
In reply to: Pintu Kumar: "Re: [perf] perf probe fails sometimes on 4.9"
Next in thread: Pintu Kumar: "Re: [perf] perf probe fails sometimes on 4.9"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]