Re: [perf] perf probe fails sometimes on 4.9

From: Pintu Kumar
Date: Mon Jan 29 2018 - 11:30:57 EST

Next message: kan . liang: "[PATCH V3 0/5] bugs fix for large PEBS mmap read and rdpmc read"
Previous message: kan . liang: "[PATCH V3 1/5] perf/x86/intel: fix event update for auto-reload"
In reply to: Masami Hiramatsu: "Re: [perf] perf probe fails sometimes on 4.9"
Next in thread: Masami Hiramatsu: "Re: [perf] perf probe fails sometimes on 4.9"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Dear Masami,

Thank you so much for your reply.
Please find some of my answers inline.

On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote:
> On Mon, 29 Jan 2018 13:40:34 +0530
> Pintu Kumar <pintu.ping@xxxxxxxxx> wrote:
>
>> Hi All,
>>
>> 'perf probe' is failing sometimes on 4.9.20 with AMD-64.
>> # perf probe --add schedule
>> schedule is out of .text, skip it.
>> Error: Failed to add events.
>>
>> If any one have come across this problem please let me know the cause.
>
> Hi Pintu,
>
> Could you run it with --vv?
>
Ok, I will send verbose output by tomorrow.

>>
>> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem?
>
> Without it, you can not probe source-level probe nor trace local variable.
>

Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20
However, I will try to manually include "-g" option during compilation.

>> However, I manually copied the vmlinux file to /boot/ directory, but
>> still it does not work.
>
> That doesn't work.
> CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo.
> Without that option, debuginfo is not generated with vmlinux.
>
>>
>> I checked upstream patches until 4.15 but could not find any clue.
>> Please let me know if there is any fixes available for this.
>
> Could you also ensure that you run perf by root user?
>

Yes I am running with root user.

My concern is sometimes it works but sometimes it fails.
I still needs to figure out, in which condition it works and which
condition it fails.
Usually I noticed that in fresh reboot case it works.

> Thank you,
>
>
>>
>>
>> Thank You!
>> Regards,
>> Pintu
>>
>>
>> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar <pintu.ping@xxxxxxxxx> wrote:
>> > Hi,
>> >
>> > ** Changed the subject now, since these issues are related to general
>> > perf commands.
>> >
>> > Following are the issues:
>> >
>> > 1) perf probe --add schedule - FAILED
>> > output:
>> > schedule is out of .text, skip it.
>> > Error: Failed to add events.
>> >
>> > what is the issue here?
>> > Sometimes it pass and sometimes it fails...
>> > Similar is the case of 'perf inject' as well.
>> >
>> > 2) perf test - 1 FAILURE
>> > 37.1: Test basic BPF filtering : FAILED!
>> > 37.2: Test BPF prologue generation : Skip
>> > 37.3: Test BPF relocation checker : Skip
>> >
>> > bpf: config program 'func=SyS_epoll_wait'
>> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null)
>> > bpf: config 'func=SyS_epoll_wait' is ok
>> > Looking at the vmlinux_path (8 entries long)
>> > Using /boot/vmlinux for symbols
>> > Could not open debuginfo. Try to use symbols.
>> > SyS_epoll_wait is out of .text, skip it.
>> > bpf_probe: failed to convert perf probe eventsFailed to add events
>> > selected by BPF
>> > test child finished with -1
>> > ---- end ----
>> > Test BPF filter subtest 0: FAILED!
>> >
>> > Looks like both 1,2 are related.
>> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux
>> > to /boot/ folder.
>> >
>> > -------
>> > Some more info:
>> >
>> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build
>> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> > unset env: KBUILD_OPTS
>> > include option is set to -nostdinc -isystem
>> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > set env: NR_CPUS=8
>> > set env: LINUX_VERSION_CODE=0x40914
>> > set env: CLANG_EXEC=/usr/bin/clang
>> > set env: CLANG_OPTIONS=-xc
>> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
>> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include
>> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated
>> > -I./include -I./arch/x86/include/uapi -I./include/uapi
>> > -I./include/generated/uapi -include ./include/linux/kconfig.h
>> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build
>> >
>> >
>> > If you have any clue about these failure please help me.
>> >
>> >
>> > Thanks,
>> > Pintu
>> >
>> >
>> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar <pintu.ping@xxxxxxxxx> wrote:
>> >> Hi,
>> >>
>> >> Thanks for your help.
>> >> Yes it was a sub version issue.
>> >>
>> >> Earlier, while building the kernel I left the SUBLEVEL as blank.
>> >>
>> >> make -j8 bindeb-pkg SUBLEVEL=20
>> >>
>> >> After passing the correct sublevel now the test is working.
>> >>
>> >> But still following are failing:
>> >>
>> >> 16: Try 'import perf' in python, checking link problems : FAILED!
>> >> 37.2: Test BPF prologue generation : FAILED!
>> >>
>> >>
>> >> This is the error I get:
>> >>
>> >> 16: Try 'import perf' in python, checking link problems :
>> >> --- start ---
>> >> test child forked, pid 7637
>> >> Traceback (most recent call last):
>> >> File "<stdin>", line 1, in <module>
>> >> ImportError: No module named perf
>> >> test child finished with -1
>> >> ---- end ----
>> >> Try 'import perf' in python, checking link problems: FAILED!
>> >>
>> >>
>> >> Looking at the vmlinux_path (8 entries long)
>> >> symsrc__init: cannot get elf header.
>> >> Failed to find the path for kernel: Invalid ELF file
>> >> bpf_probe: failed to convert perf probe eventsFailed to add events
>> >> selected by BPF
>> >> test child finished with -1
>> >> ---- end ----
>> >> Test BPF filter subtest 1: FAILED!
>> >>
>> >>
>> >>
>> >> Thanks,
>> >> Pintu
>> >>
>> >>
>> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) <wangnan0@xxxxxxxxxx> wrote:
>> >>>
>> >>>
>> >>> On 2018/1/23 20:37, Pintu Kumar wrote:
>> >>>>
>> >>>> Hi All,
>> >>>>
>> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the
>> >>>> kernel version 4.9.20.
>> >>>> I have installed several others packages including: clang, llvm
>> >>>>
>> >>>> But, when I run 'perf test' I get some FAILURE.
>> >>>> Specially, 'perf test LLVM' is failing.
>> >>>>
>> >>>> Please check the below error logs:
>> >>>>
>> >>>> ----------------------------------------------------
>> >>>> # perf test LLVM
>> >>>> 35: Test LLVM searching and compiling :
>> >>>> 35.1: Basic BPF llvm compiling test : FAILED!
>> >>>> 35.2: Test kbuild searching : Skip
>> >>>> 35.3: Compile source for BPF prologue generation test : Skip
>> >>>> 35.4: Compile source for BPF relocation test : Skip
>> >>>> ----------------------------------------------------
>> >>>>
>> >>>> When I run with -v I get this:
>> >>>>
>> >>>> -----------------------------------------------------
>> >>>> # perf test -v LLVM
>> >>>> 35: Test LLVM searching and compiling :
>> >>>> 35.1: Basic BPF llvm compiling test :
>> >>>> --- start ---
>> >>>> test child forked, pid 3304
>> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> >>>> WARNING: unable to get correct kernel building directory.
>> >>>> Hint: Set correct kbuild directory using 'kbuild-dir' option in [llvm]
>> >>>> section of ~/.perfconfig or set it to "" to suppress kbuild
>> >>>> detection.
>> >>>>
>> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64'
>> >>>>
>> >>>>
>> >>>> int _version SEC("version") = LINUX_VERSION_CODE;
>> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS
>> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
>> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
>> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o
>> >>>> -
>> >>>> libbpf: 2129190-4a0 doesn't provide kernel version
>> >>>> Failed to parse test case 'Basic BPF llvm compiling test'
>> >>>> test child finished with -1
>> >>>> ---- end ----
>> >>>> Test LLVM searching and compiling subtest 0: FAILED!
>> >>>> -----------------------------------------------------
>> >>>>
>> >>>>
>> >>>> Basic BPF LLVM compiling test is failing.
>> >>>> It reports that bpf could not able to read the kernel version from uname.
>> >>>> Is it because of this that 'perf test LLVM' is failing ?
>> >>>>
>> >>>> My uname says:
>> >>>> 4.9-xxxx-amd-x86-64
>> >>>
>> >>>
>> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see
>> >>> tools/perf/fetch_kernel_version . If your distro uses another method to
>> >>> report
>> >>> kernel version number (from your uname string, sublevel can't be
>> >>> determined),
>> >>> please help us improve that function like what we have done for ubuntu
>> >>> (checking /proc/version_signature).
>> >>>
>> >>>
>> >>>
>> >>>> My kernel version is: 4.9.20 (from Makefile)
>> >>>>
>> >>>> So, I dont think there is any problem with kernel version.
>> >>>>
>> >>>> If any body have come across this problem please let me know how to
>> >>>> resolve this issue.
>> >>>>
>> >>>>
>> >>>>
>> >>>> Thank You!
>> >>>>
>> >>>> Regards,
>> >>>> Pintu
>> >>>
>> >>>
>> >>>
>
>
> --
> Masami Hiramatsu <mhiramat@xxxxxxxxxx>

Next message: kan . liang: "[PATCH V3 0/5] bugs fix for large PEBS mmap read and rdpmc read"
Previous message: kan . liang: "[PATCH V3 1/5] perf/x86/intel: fix event update for auto-reload"
In reply to: Masami Hiramatsu: "Re: [perf] perf probe fails sometimes on 4.9"
Next in thread: Masami Hiramatsu: "Re: [perf] perf probe fails sometimes on 4.9"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]