Re: [PATCH][RFC] tracing: Enable tracepoints via module parameters

[Dan Williams]

On Mon, Apr 19, 2021 at 3:11 PM Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
>
> On Mon, 19 Apr 2021 21:54:13 +0000
> "Williams, Dan J" <dan.j.williams@xxxxxxxxx> wrote:
>
> > [ drop Rusty, add Jessica and Emmanuel ]
>
> Probably could have kept Jessica on as she's the module maintainer.

Oh, you misread, I swapped out Rusty for Jessica on the Cc.

>
> >
> > On Wed, 2013-08-14 at 23:32 -0400, Steven Rostedt wrote:
> > > On Thu, 15 Aug 2013 11:32:10 +0930
>
> Wow, this is coming back from the dead! No thread rests in peace!

No one can escape the all seeing eye of lore, and it makes it so easy
to reply to ancient stuff. I thought it useful to kick the zombie
because this thread is the first thing that comes up in the Google
search of:

"enable tracepoint at module load"

>
> > > Rusty Russell <rusty@xxxxxxxxxxxxxxx> wrote:
> > >
> > > > Steven Rostedt <rostedt@xxxxxxxxxxx> writes:
> > > > > But the thing about this that bothers me is that there's no way
> > > > > to say,
> > > > > "Enable all tracepoints in this module on load". I would like a
> > > > > way to
> > > > > do that, but I don't know of a way to do that without modifying
> > > > > the
> > > > > module code. Have any ideas? Basically, I would love to have:
> > > > >
> > > > > insmod foo tracepoints=all
> > > > >
> > > > > or something and have all tracepoints enabled.
> > > >
> > > > "without modifying the module code"?  Why?  The code isn't that
> > > > scary,
> > > > and this seems useful.
> > >
> > > I'm not afraid of the code, I'm afraid of you ;-) I hear you have
> > > killer puppies.
> > >
> > >
> > > OK, then when I get some time, I may cook something up.
>
>
> "when I get some time" HAHAHAHAHAHAH!!!!  That was what? 8 years ago!

free time coming up... any day now.

>
> > >
> > > Thanks,
> > >
> > > -- Steve
> >
> > Revive an old thread...
>
> I'll say!
>
> >
> > Steven, did you ever end up with a solution to the "enable tracing at
> > module load" problem?
>
> For tracepoints, no. For function tracing, yes!
>
> >
> > I see some people getting admonished to use ftrace over dev_dbg() [1],
> > but one of the features that keeps dev_dbg() proliferating is its
> > generic "$mod_name.dyndbg=" module parameter support for selective
> > debug enabling at boot / module-load.
> >
> > It would be useful to be able to do
> > /sys/kernel/debug/dynamic_debug/control enabling for tracepoints, but
> > also module::function_name patterns for "got here" style debugging. I'd
> > be happy to help with this, but wanted to understand where you left
> > things.
> >
> > [1]: https://lore.kernel.org/linux-wireless/YHRFy3aq%2FgB7Vde6@xxxxxxxxx/
>
> I don't think I did anything with trace events, I'll have to dig deeper.
> But today you have this:
>
>  # cd /sys/kernel/tracing
>
>  # rmmod bridge
>
>  # echo ':mod:bridge' > set_ftrace_filter
>
>  # cat set_ftrace_filter
> :mod:bridge
>
>  # echo function > current_tracer
>
>  # cat trace
> # tracer: function
> #
> # entries-in-buffer/entries-written: 0/0   #P:8
> #
> #                                _-----=> irqs-off
> #                               / _----=> need-resched
> #                              | / _---=> hardirq/softirq
> #                              || / _--=> preempt-depth
> #                              ||| /     delay
> #           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
> #              | |         |   ||||      |         |
>
>  # modprobe bridge
>
>  # cat set_ftrace_filter
> br_switchdev_event [bridge]
> br_device_event [bridge]
> br_net_exit [bridge]
> br_boolopt_get [bridge]
> br_boolopt_multi_get [bridge]
> br_opt_toggle [bridge]
> br_boolopt_toggle [bridge]
> br_boolopt_multi_toggle [bridge]
> br_dev_set_multicast_list [bridge]
> br_get_link_ksettings [bridge]
> [..]
>
>  # cat trace
> # tracer: function
> #
> # entries-in-buffer/entries-written: 10/10   #P:8
> #
> #                                _-----=> irqs-off
> #                               / _----=> need-resched
> #                              | / _---=> hardirq/softirq
> #                              || / _--=> preempt-depth
> #                              ||| /     delay
> #           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
> #              | |         |   ||||      |         |
>         modprobe-2364    [006] .... 12929.869510: br_init <-do_one_initcall
>         modprobe-2364    [006] .... 12929.869513: br_fdb_init <-br_init
>         modprobe-2364    [006] .... 12929.869522: br_device_event <-call_netdevice_register_net_notifiers
>         modprobe-2364    [006] .... 12929.869522: br_device_event <-call_netdevice_register_net_notifiers
>         modprobe-2364    [006] .... 12929.869522: br_device_event <-call_netdevice_register_net_notifiers
>         modprobe-2364    [006] .... 12929.869522: br_device_event <-call_netdevice_register_net_notifiers
>         modprobe-2364    [006] .... 12929.869523: br_device_event <-call_netdevice_register_net_notifiers
>         modprobe-2364    [006] .... 12929.869523: br_device_event <-call_netdevice_register_net_notifiers
>         modprobe-2364    [006] .... 12929.869524: br_netlink_init <-br_init
>         modprobe-2364    [006] .... 12929.869524: br_mdb_init <-br_netlink_init
>
>
> So yes, function tracing now allows setting a filter to trace only the
> functions for a given module, and if that module is not yet loaded, it
> stores the filter until it is.

Ah, thanks for the pointer. So if I wanted to convert a kernel command like:

libnvdimm.dyndbg

...it would be something like:

ftrace=function ftrace_filter=:mod:libnvdimm

...and then "cat /sys/kernel/tracing/trace" instead of "dmesg" to
retrieve... assuming only "got here" style debug was being attempted.

> To do something similar for tracepoints, I think we still need to add it as
> a module parameter.

The dev_dbg() filter language is attractive, it's too bad
trace_printk() has such a high runtime cost as combining dynamic-debug
and tracing would seem to be a panacea.