Re: [PATCH] perf probe: Ignore tail calls to probed functions

From: Naveen N. Rao
Date: Thu Apr 30 2015 - 11:00:14 EST

Next message: Jonathan McDowell: "Re: Audio crackles with 4.1-rc1"
Previous message: Julian Calaby: "Re: [linux-sunxi] [PATCH 3/3] MTD: m25p80: Add option to limit SPI transfer size."
In reply to: Masami Hiramatsu: "Re: [PATCH] perf probe: Ignore tail calls to probed functions"
Next in thread: Masami Hiramatsu: "Re: [PATCH] perf probe: Ignore tail calls to probed functions"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

[CC'ing Srikar]

On 2015/04/30 10:06PM, Masami Hiramatsu wrote:
> On 2015/04/30 20:42, Naveen N. Rao wrote:
> > perf probe currently errors out if there are any tail calls to probed
> > functions:
> >
> > [root@rhel71be]# perf probe do_fork
> > Failed to find probe point in any functions.
> > Error: Failed to add events.
> >
> > Fix this by teaching perf to ignore tail calls.
>
> Hmm, why just ignore it, can't we use it?
> Also, could you show us how the result will be changed with this?

Without patch:

[root@rhel71be perf]# ./perf probe -v do_fork
probe-definition(0): do_fork symbol:do_fork file:(null) line:0 offset:0
return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
symsrc__init: build id mismatch for /boot/vmlinux.
Using /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux for
symbols
Open Debuginfo file:
/usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux
Try to find probe point from debuginfo.
found inline addr: 0xc0000000000bb9b0
Probe point found: do_fork+0
found inline addr: 0xc0000000000bbe20
Probe point found: kernel_thread+48
found inline addr: 0xc0000000000bbe5c
Probe point found: sys_fork+28
found inline addr: 0xc0000000000bbfac
Probe point found: sys_vfork+44
found inline addr: 0xc0000000000bc27c
Failed to find probe point in any functions.
An error occurred in debuginfo analysis (-2).
Error: Failed to add events. Reason: No such file or directory (Code:
-2)

With patch:

[root@rhel71be perf]# ./perf probe -v do_fork
probe-definition(0): do_fork symbol:do_fork file:(null) line:0 offset:0
return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (7 entries long)
symsrc__init: build id mismatch for /boot/vmlinux.
Using /usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux for
symbols
Open Debuginfo file:
/usr/lib/debug/lib/modules/3.10.0-201.el7.ppc64/vmlinux
Try to find probe point from debuginfo.
found inline addr: 0xc0000000000bb9b0
Probe point found: do_fork+0
found inline addr: 0xc0000000000bbe20
Probe point found: kernel_thread+48
found inline addr: 0xc0000000000bbe5c
Probe point found: sys_fork+28
found inline addr: 0xc0000000000bbfac
Probe point found: sys_vfork+44
found inline addr: 0xc0000000000bc27c
Ignoring tail call from SyS_clone
Found 4 probe_trace_events.
Opening /sys/kernel/debug/tracing/kprobe_events write=1
No kprobe blacklist support, ignored
Added new events:
Writing event: p:probe/do_fork _text+768432
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)

[Ignore the error about failure to write event - this kernel is missing
a patch to resolve _text properly]

The reason to ignore tail calls is that the address does not belong to
any function frame. In the example above, the address in SyS_clone is
0xc0000000000bc27c, but looking at the debug-info:

<1><830081>: Abbrev Number: 133 (DW_TAG_subprogram)
<830083> DW_AT_external : 1
<830083> DW_AT_name : (indirect string, offset: 0x3cea3): SyS_clone
<830087> DW_AT_decl_file : 7
<830088> DW_AT_decl_line : 1689
<83008a> DW_AT_prototyped : 1
<83008a> DW_AT_type : <0x8110eb>
<83008e> DW_AT_low_pc : 0xc0000000000bc270
<830096> DW_AT_high_pc : 0xc
<83009e> DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa)
<8300a0> DW_AT_GNU_all_call_sites: 1
<8300a0> DW_AT_sibling : <0x830178>
<snip>
<3><830147>: Abbrev Number: 125 (DW_TAG_GNU_call_site)
<830148> DW_AT_low_pc : 0xc0000000000bc27c
<830150> DW_AT_GNU_tail_call: 1
<830150> DW_AT_abstract_origin: <0x82e7e1>

The frame ends at 0xc0000000000bc27c. I suppose this is why this
particular call is a "tail" call. FWIW, systemtap seems to ignore these
as well and requires users to explicitly place probes at these call
sites if necessary. I print out the caller so that users know.

>
> Thank you,

Thanks for the review!
- Naveen

>
> >
> > Signed-off-by: Naveen N. Rao <naveen.n.rao@xxxxxxxxxxxxxxxxxx>
> > ---
> > I'm seeing this with RHEL7 kernels on ppc64. The specific example in this case is do_fork:
> >
> > <1><82e7e1>: Abbrev Number: 118 (DW_TAG_subprogram)
> > <82e7e2> DW_AT_external : 1
> > <82e7e2> DW_AT_name : (indirect string, offset: 0x3dfe0): do_fork
> > <82e7e6> DW_AT_decl_file : 7
> > <82e7e7> DW_AT_decl_line : 1583
> > <82e7e9> DW_AT_prototyped : 1
> > <82e7e9> DW_AT_type : <0x8110eb>
> > <82e7ed> DW_AT_inline : 1 (inlined)
> > <82e7ee> DW_AT_sibling : <0x82e878>
> >
> > <snip>
> >
> > <1><830081>: Abbrev Number: 133 (DW_TAG_subprogram)
> > <830083> DW_AT_external : 1
> > <830083> DW_AT_name : (indirect string, offset: 0x3cea3): SyS_clone
> > <830087> DW_AT_decl_file : 7
> > <830088> DW_AT_decl_line : 1689
> > <83008a> DW_AT_prototyped : 1
> > <83008a> DW_AT_type : <0x8110eb>
> > <83008e> DW_AT_low_pc : 0xc0000000000bc270
> > <830096> DW_AT_high_pc : 0xc
> > <83009e> DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa)
> > <8300a0> DW_AT_GNU_all_call_sites: 1
> > <8300a0> DW_AT_sibling : <0x830178>
> >
> > <snip>
> >
> > <3><830147>: Abbrev Number: 125 (DW_TAG_GNU_call_site)
> > <830148> DW_AT_low_pc : 0xc0000000000bc27c
> > <830150> DW_AT_GNU_tail_call: 1
> > <830150> DW_AT_abstract_origin: <0x82e7e1>
> >
> >
> > - Naveen
> >
> > tools/perf/util/dwarf-aux.c | 37 +++++++++++++++++++++++++++++++++++++
> > tools/perf/util/dwarf-aux.h | 4 ++++
> > tools/perf/util/probe-finder.c | 12 +++++++++---
> > 3 files changed, 50 insertions(+), 3 deletions(-)
> >
> > diff --git a/tools/perf/util/dwarf-aux.c b/tools/perf/util/dwarf-aux.c
> > index c34e024..851a76f 100644
> > --- a/tools/perf/util/dwarf-aux.c
> > +++ b/tools/perf/util/dwarf-aux.c
> > @@ -417,6 +417,43 @@ struct __addr_die_search_param {
> > Dwarf_Die *die_mem;
> > };
> >
> > +static int __die_search_func_tail_cb(Dwarf_Die *fn_die, void *data)
> > +{
> > + struct __addr_die_search_param *ad = data;
> > + Dwarf_Addr addr = 0;
> > +
> > + if (dwarf_tag(fn_die) == DW_TAG_subprogram &&
> > + !dwarf_highpc(fn_die, &addr) &&
> > + addr == ad->addr) {
> > + memcpy(ad->die_mem, fn_die, sizeof(Dwarf_Die));
> > + return DWARF_CB_ABORT;
> > + }
> > + return DWARF_CB_OK;
> > +}
> > +
> > +/**
> > + * die_find_tailfunc - Search for a non-inlined function with tail call at
> > + * given address
> > + * @cu_die: a CU DIE which including @addr
> > + * @addr: target address
> > + * @die_mem: a buffer for result DIE
> > + *
> > + * Search for a non-inlined function DIE with tail call at @addr. Stores the
> > + * DIE to @die_mem and returns it if found. Returns NULL if failed.
> > + */
> > +Dwarf_Die *die_find_tailfunc(Dwarf_Die *cu_die, Dwarf_Addr addr,
> > + Dwarf_Die *die_mem)
> > +{
> > + struct __addr_die_search_param ad;
> > + ad.addr = addr;
> > + ad.die_mem = die_mem;
> > + /* dwarf_getscopes can't find subprogram. */
> > + if (!dwarf_getfuncs(cu_die, __die_search_func_tail_cb, &ad, 0))
> > + return NULL;
> > + else
> > + return die_mem;
> > +}
> > +
> > /* die_find callback for non-inlined function search */
> > static int __die_search_func_cb(Dwarf_Die *fn_die, void *data)
> > {
> > diff --git a/tools/perf/util/dwarf-aux.h b/tools/perf/util/dwarf-aux.h
> > index af7dbcd..c9278ed 100644
> > --- a/tools/perf/util/dwarf-aux.h
> > +++ b/tools/perf/util/dwarf-aux.h
> > @@ -82,6 +82,10 @@ extern Dwarf_Die *die_find_child(Dwarf_Die *rt_die,
> > extern Dwarf_Die *die_find_realfunc(Dwarf_Die *cu_die, Dwarf_Addr addr,
> > Dwarf_Die *die_mem);
> >
> > +/* Search a non-inlined function with tail call at given address */
> > +Dwarf_Die *die_find_tailfunc(Dwarf_Die *cu_die, Dwarf_Addr addr,
> > + Dwarf_Die *die_mem);
> > +
> > /* Search the top inlined function including given address */
> > extern Dwarf_Die *die_find_top_inlinefunc(Dwarf_Die *sp_die, Dwarf_Addr addr,
> > Dwarf_Die *die_mem);
> > diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
> > index b5bf9d5..4cb461e 100644
> > --- a/tools/perf/util/probe-finder.c
> > +++ b/tools/perf/util/probe-finder.c
> > @@ -660,9 +660,15 @@ static int call_probe_finder(Dwarf_Die *sc_die, struct probe_finder *pf)
> > /* If not a real subprogram, find a real one */
> > if (!die_is_func_def(sc_die)) {
> > if (!die_find_realfunc(&pf->cu_die, pf->addr, &pf->sp_die)) {
> > - pr_warning("Failed to find probe point in any "
> > - "functions.\n");
> > - return -ENOENT;
> > + if (die_find_tailfunc(&pf->cu_die, pf->addr, &pf->sp_die)) {
> > + pr_warning("Ignoring tail call from %s\n",
> > + dwarf_diename(&pf->sp_die));
> > + return 0;
> > + } else {
> > + pr_warning("Failed to find probe point in any "
> > + "functions.\n");
> > + return -ENOENT;
> > + }
> > }
> > } else
> > memcpy(&pf->sp_die, sc_die, sizeof(Dwarf_Die));
> >
>
>
> --
> Masami HIRAMATSU
> Linux Technology Research Center, System Productivity Research Dept.
> Center for Technology Innovation - Systems Engineering
> Hitachi, Ltd., Research & Development Group
> E-mail: masami.hiramatsu.pt@xxxxxxxxxxx
>

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Jonathan McDowell: "Re: Audio crackles with 4.1-rc1"
Previous message: Julian Calaby: "Re: [linux-sunxi] [PATCH 3/3] MTD: m25p80: Add option to limit SPI transfer size."
In reply to: Masami Hiramatsu: "Re: [PATCH] perf probe: Ignore tail calls to probed functions"
Next in thread: Masami Hiramatsu: "Re: [PATCH] perf probe: Ignore tail calls to probed functions"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]