[PATCH 0/2] ftrace: Fix subops accounting

[Steven Rostedt]

A fix [1] came in that fixed the notrace_filter side of the subops processing
of the function graph tracer. When I started testing that fix, I discovered
that the many more functions were being enabled than were being traced.

The function graph infrastructure uses ftrace to hook to functions. It has
a single ftrace_ops to manage all the users of function graph. Each
individual user (tracing, bpf, fprobes, etc) has its own ftrace_ops to
track the functions it will have its callback called from. These
ftrace_ops are "subops" to the main ftrace_ops of the function graph
infrastructure.

Each ftrace_ops has a filter_hash and a notrace_hash that is defined as:

  Only trace functions that are in the filter_hash but not in the
  notrace_hash.

If the filter_hash is empty, it means to trace all functions.
If the notrace_hash is empty, it means do not disable any function.

The function graph main ftrace_ops needs to be a superset containing all
the functions to be traced by all the subops it has. The algorithm to
perform this merge was incorrect. It was merging the filter_hashes
of all the subops and taking the intersect of all the notrace_hashes
of the subops. But by taking the intersect of all the notrace_hashes
it ignored how those notrace_hashes are dependent on the associated
filter_hashes of each individual subops.

Instead, modify the algorithm to be a bit simpler and correct.

First, when adding a new subops, do not add the notrace_hash if the
filter_hash is not empty. Instead, just add the functions that are in the
filter_hash of the subops but not in the notrace_hash of the subops into the
main ops filter_hash. There's no reason to add anything to the main ops
notrace_hash for this case.

The notrace_hash of the main ops should only be non empty iff all subops
filter_hashes are empty (meaning to trace all functions) and all subops
notrace_hashes have the same functions.

That is, the main ops notrace_hash is empty if any subops filter_hash is
non empty.

The main ops notrace_hash only has content in it if all subops
filter_hashes are empty, and the content are only functions that intersect
all the subops notrace_hashes. If any subops notrace_hash is empty, then
so is the main ops notrace_hash.

[1] https://lore.kernel.org/all/20250408160258.48563-1-andybnac@xxxxxxxxx/

Steven Rostedt (2):
      ftrace: Fix accounting of subop hashes
      tracing/selftest: Add test to better test subops filtering of function graph

----
 kernel/trace/ftrace.c                              | 314 ++++++++++++---------
 .../ftrace/test.d/ftrace/fgraph-multi-filter.tc    | 177 ++++++++++++
 2 files changed, 354 insertions(+), 137 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-multi-filter.tc