Re: [PATCH perf-tools-next v9 0/6] perf ftrace: Support inlined functions and display enhancements in function graph tracer

[Aaron Tomlin]

On Wed, Oct 21, 2026 at 09:36:59AM -0700, Ian Rogers wrote:
> On Tue, Oct 20, 2026 at 7:33 PM Aaron Tomlin <atomlin@xxxxxxxxxxx> wrote:
> >
> > The Linux kernel's function graph tracer operates at the machine
> > instruction level via compiler instrumentation
> > (-fpatchable-function-entry), recording entry and return events for
> > physical function calls. Consequently, functions inlined by the compiler
> > are invisible in the resulting call graph, as no discrete call or return
> > instructions are emitted for them. While technically expected, this
> > omission frequently obscures the logical execution flow when analysing
> > kernel subsystems heavily reliant upon inlining (such as the scheduler,
> > memory management, locking primitives, and RCU).
> >
> > When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the
> > funcgraph-retaddr trace option records the caller's return address on each
> > function entry, manifested in the trace stream as a comment
> > (e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug
> > information from the kernel image (vmlinux) utilising libdw, perf ftrace
> > resolves this return address to its inlined callchain and synthesises the
> > intermediate inlined frames directly into the streamed call graph.
> >
> > This patch series addresses these challenges through six focused commits:
> >
> >     1.  Stack frame optimisation in __cmd_ftrace()
> >
> >         Replaces the 4096-byte stack array in __cmd_ftrace() with
> >         dynamic heap allocation alongside struct strbuf streaming,
> >         reducing the stack frame footprint by over 96% (down to
> >         ~160 bytes) whilst ensuring complete line-delimited records,
> >         explicit NUL termination, and robust error unwind and EOF
> >         handling.
> >
> >     2.  Return address comment filtering
> >
> >         Introduces ftrace_parse_retaddr() and ftrace_filter_retaddr()
> >         with --filter-retaddr and --graph-opts filter-retaddr to strip
> >         caller comments from trace output, accompanied by an initial
> >         unit test suite in tools/perf/tests/ftrace.c.
> >
> >     3.  Preparatory vmlinux lookup helper
> >
> >         Introduces vmlinux_path__find() in tools/perf/util/symbol.c and
> >         symbol.h to search standard vmlinux paths and return the first
> >         accessible kernel image, providing a generic helper across
> >         tools/perf and avoiding hardcoded access() checks in tests.
> >
> >     4.  Inlined function call graph reconstruction
> >
> >         Introduces --inline and -k/--vmlinux, dynamically synthesising
> >         inlined frames marked with /* (inline) */. Correctly matches
> >         bare closing braces when funcgraph-tail is disabled, clamps stack
> >         depth to prevent indentation drift, and expands the automated test
> >         suite to verify inline resolution, comment filtering, and stack
> >         saturation recovery utilising vmlinux_path__find().
> >
> >     5.  Execution duration suppression
> >
> >         Introduces --graph-opts noduration (accepting noduration=[0|1])
> >         to toggle the kernel's funcgraph-duration tracing option,
> >         omitting the duration column when the user is focused purely on
> >         call graph topology and inlined hierarchy. Rejects noduration
> >         when invoked under 'perf ftrace profile'.
> >
> >     6.  Extraneous function filtering in function graph tracer
> >
> >         Introduces userspace call graph filtering when targeted
> >         tracing (-G or --graph-funcs) is active. Suppresses transient
> >         unrelated entries arriving before filter activation as well
> >         as detached callee trees (such as ring-buffer wakeups)
> >         originating outside the target hierarchy. Dynamically resises
> >         the synthetic call stack to eliminate saturation edge cases and
> >         accurately suppresses detached subtrees without orphan exit name
> >         collisions.
> >
> > Synthesised inlined functions are rendered with an explicit /* (inline) */
> > annotation at both entry and exit:
> >
> >     # tracer: function_graph
> >     #
> >     # CPU               FUNCTION CALLS
> >     # |                 |   |   |   |
> >      0) enqueue_entity() {
> >      0)   update_cfs_group() { /* (inline) */
> >      0)     calc_concur_shares() {
> >      0)       tg_cpus() {
> >      0)         cpuset_num_cpus() {
> >      0)           class_rcu_constructor() { /* (inline) */
> >      0)             rcu_read_lock() { /* (inline) */
> >      0)               __rcu_read_lock();
> >      0)             } /* rcu_read_lock (inline) */
> >      0)           } /* class_rcu_constructor (inline) */
> >      0)           cgroup_e_css();
> >      0)           class_rcu_destructor() { /* (inline) */
> >      0)             rcu_read_unlock() { /* (inline) */
> >      0)               __rcu_read_unlock();
> >      0)             } /* rcu_read_unlock (inline) */
> >      0)           } /* class_rcu_destructor (inline) */
> >      0)         }
> >      0)       }
> >      0)       __calc_smp_shares();
> >      0)     reweight_entity();
> >      0)   } /* update_cfs_group (inline) */
> >      0) }
> 
> Lots of tests, Sashiko is happy and a useful feature!  I'm not sure
> "--filter-retaddr" is the best name for an option to disable comments
> on return addresses, but naming things is hard.
> 
> Tested-by: Ian Rogers <irogers@xxxxxxxxxx>
> Reviewed-by: Ian Rogers <irogers@xxxxxxxxxx>
> 
> Thanks!
> Ian

Indeed the name is not the best :)

Thank you Ian!
-- 
Aaron Tomlin