Re: [PATCH perf-tools-next v12 06/10] perf ftrace: Support display of inlined functions in function graph tracer

[Namhyung Kim]

On Sun, Oct 25, 2026 at 10:57:31PM -0400, Aaron Tomlin wrote:
> The Linux kernel's function graph tracer operates at the machine
> instruction level via compiler instrumentation
> (-fpatchable-function-entry), recording entry and return events for
> physical function calls. Consequently, functions inlined by the compiler
> are invisible in the resulting call graph, as no discrete call or return
> instructions are emitted for them. While technically expected, this
> omission frequently obscures the logical execution flow when analysing
> kernel subsystems heavily reliant upon inlining.
> 
> Address this by introducing the --inline option to perf ftrace for the
> function_graph tracer.
> 
> When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the
> funcgraph-retaddr trace option records the caller's return address on
> each function entry, manifested in the trace stream as a comment
> (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug
> information from the kernel image (vmlinux) utilising libdw, perf ftrace
> resolves this return address to its inlined callchain and synthesises the
> intermediate inlined frames directly into the streamed call graph.
> 
> Key aspects of this implementation:
> 
>     1.  DWARF inline resolution
> 
>         Implement ftrace_resolve_inlines() using libdw to locate inlined
>         subroutines at the instruction address corresponding to the
>         caller's return address.
> 
>         Implement perf_ftrace__setup_inlines() and
>         perf_ftrace__cleanup_inlines() to initialise machine symbol
>         tracking, load vmlinux symbols, and verify kernel
>         CONFIG_FUNCTION_GRAPH_RETADDR support.
> 
>     2.  Inline frame synthesis
> 
>         In ftrace_process_fgraph_line(), when --inline is active:
>             - Interrogate return addresses using ftrace_parse_retaddr()
>               and resolve inlines via ftrace_resolve_inlines().
> 
>             - Identify common inlined ancestors on the tracked stack and
>               unwind inactive frames.
> 
>             - Open new inlined frames with /* (inline) */ and indent by
>               inlined_depth * 2.
> 
>             - On function exit, close matching inlined frames with
>               } /* func (inline) */.
> 
>     3.  Command-line options and documentation
> 
>         Add --inline and -k/--vmlinux options in
>         tools/perf/builtin-ftrace.c and document their usage in
>         tools/perf/Documentation/perf-ftrace.txt.
> 
> Tested-by: Ian Rogers <irogers@xxxxxxxxxx>
> Reviewed-by: Ian Rogers <irogers@xxxxxxxxxx>
> Signed-off-by: Aaron Tomlin <atomlin@xxxxxxxxxxx>
> ---
[SNIP]
> @@ -664,11 +834,16 @@ static void extract_exit_func_name(const char *str, char *buf, size_t size)
>   * @cpu: Pointer to store the CPU number
>   *
>   * Locate the context columns that precede the function call column of a
> - * function_graph trace record.
> + * function_graph trace record. Depending on the funcgraph-* options set
> + * up by perf, a record looks like:
> + *
> + *   [ abstime |]  CPU)  [ comm-pid |]  [ duration |]  func
> + *
> + * The abstime and comm-pid columns are only present with --graph-opts
> + * verbose.
>   *
>   * Return: A pointer to the last character of the context columns (the final
> - * '|' delimiter, or the CPU ')' when no delimiter follows it). Returns NULL
> - * if @line is not a trace record.
> + * '|' delimiter). Returns NULL if @line is not a trace record.
>   */

It'd be nice if you could move the comments to the original commit that
introduced the function.


>  static const char *find_func_column(const struct perf_ftrace *ftrace,
>  				    const char *line, int *cpu)
[SNIP]
> diff --git a/tools/perf/util/ftrace.h b/tools/perf/util/ftrace.h
> index 45f5090d0115..7b9e0919ae8d 100644
> --- a/tools/perf/util/ftrace.h
> +++ b/tools/perf/util/ftrace.h
> @@ -42,6 +42,8 @@ struct perf_ftrace {
>  	int			graph_verbose;
>  	int			graph_thresh;
>  	int			graph_tail;
> +	bool			use_inline;
> +	struct machine		*machine;
>  };
>  
>  struct filter_entry {
> @@ -96,44 +98,18 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
>  
>  #endif  /* HAVE_BPF_SKEL */
>  
> -#define FTRACE_MAX_CPUS 4096
> -#define FTRACE_MAX_STACK 256
> -#define FTRACE_PID_HASH_BITS 10
> -#define FTRACE_PID_HASH_SIZE (1 << FTRACE_PID_HASH_BITS)
> +struct machine;
>  
> -#define FTRACE_IDLE_PID(cpu)       ((pid_t)(-1 - (cpu)))
> -#define FTRACE_UNKNOWN_PID(cpu)    ((pid_t)(-1 - FTRACE_MAX_CPUS - (cpu)))
> -
> -struct ftrace_stack_frame {
> -	char *name;
> -	bool inlined;
> -	int indent;
> -	int raw_indent;
> -	u64 seq;
> -};
> -
> -struct ftrace_indent_watermark {
> -	u64 seq;
> -	int raw_indent;
> -};
> -
> -struct ftrace_task_state {
> -	struct hlist_node	node;
> -	pid_t			pid;
> -	struct ftrace_stack_frame *stack;
> -	int			max_stack;
> -	int			depth;
> -	int			inlined_depth;
> -	int			overflow_depth;
> -	u64			seq;
> -	struct ftrace_indent_watermark *watermarks;
> -	int			max_watermarks;
> -	int			nr_watermarks;
> -};

Hmm.. probably better to define the structs in the .c file in the first
place?

Thanks,
Namhyung


> +#define FTRACE_MAX_INLINES 32
>  
>  bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
>  void ftrace_filter_retaddr(char *str);
> -void ftrace_cpu_states__clear(void);
> +int ftrace_resolve_inlines(struct machine *machine,
> +			   const char *sym_name, u64 offset,
> +			   const char **inlined_names, int max_inlines);
>  int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
> +void ftrace_cpu_states__clear(void);
> +int perf_ftrace__setup_inlines(struct perf_ftrace *ftrace);
> +void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace);
>  
>  #endif  /* __PERF_FTRACE_H__ */
> -- 
> 2.55.0
>