Re: [RFC PATCH -tip 0/5] kprobes: Abolish jprobe APIs

[Jonathan Corbet]

On Fri, 6 Oct 2017 11:34:30 -0400
Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:

> On Fri, 6 Oct 2017 13:49:59 +0900
> Masami Hiramatsu <mhiramat@xxxxxxxxxx> wrote:
> 
> > Steve, could you write a documentation how to use ftrace callback?
> > I think I should update the Documentation/kprobes.txt so that jprobe
> > user can easily migrate on that.  
> 
> I decided to do this now. Here's a first draft. What do you think?

An overall request: this document is already 99% in the RST format.  It
would be nice to just do it in RST to begin with and save somebody the
effort of finishing the job later.

> -- Steve
> 
> 		Using ftrace to hook to functions
> 		=================================
> 
> Copyright 2017 VMware Inc.
>    Author:   Steven Rostedt <srostedt@xxxxxxxxxxx>
>   License:   The GNU Free Documentation License, Version 1.2
>                (dual licensed under the GPL v2)
> 
> Written for: 4.14

Experience says that strings like this go obsolete in a hurry, so they
don't really reflect the state of the document.  The git history tends to
be rather more reliable.

> Introduction
> ------------
> 
> The ftrace infrastructure was originially created to attach hooks to the
> beginning of functions in order to record and trace the flow of the kernel.
> But hooks to the start of a function can have other use cases. Either
> for live kernel patching, or for security monitoring. This document describes
> how to use ftrace to implement your own function hooks.
> 
> 
> The ftrace context
> ==================
> 
> WARNING: The ability to add a callback to almost any function within the
> kernel comes with risks. A callback can be called from any context
> (normal, softirq, irq, and NMI). Callbacks can also be called just before
> going to idle, during CPU bring up and takedown, or going to user space.
> This requires extra care to what can be done inside a callback. A callback
> can be called outside the protective scope of RCU.
> 
> The ftrace infrastructure has some protections agains recursions and RCU
> but one must still be very careful how they use the callbacks.
> 
> 
> The ftrace_ops structure
> ========================
> 
> To register a function callback, a ftrace_ops is required.

It would be nice to say which file should be #include'd to get these
declarations. 

> This structure
> is used to tell ftrace what function should be called as the callback
> as well as what protections the callback will perform and not require
> ftrace to handle.
> 
> There are only two fields that are needed to be set when registering
> an ftrace_ops with ftrace. The rest should be NULL.
> 
> struct ftrace_ops ops = {
>        .func			= my_callback_func,
>        .flags			= MY_FTRACE_FLAGS
>        .private			= any_private_data_structure,
> };
> 
> Both .flags and .private are optional. Only .func is required.

So...only two fields are needed, but three are shown, and only one is
required.  Dumb people like me are easily confused by such things...

> To enable tracing call:
> 
>   register_ftrace_function(&ops);
> 
> To disable tracing call:
> 
>   unregister_ftrace_function(@ops);

I would presume that the callback can be called immediately, perhaps even
before register_ftrace_function() returns, right?  It never hurts to warn
readers that they need to be ready from the outset.

> 
> The callback function
> =====================
> 
> The prototype of the callback function is as follows (as of v4.14):
> 
>  void callback_func(unsigned long ip, unsigned long parent_ip,
> 		    struct ftrace_ops *op, struct pt_regs *regs);
> 
> @ip - This is the instruction pointer of the function that is being traced.
>       (where the fentry or mcount is within the function)
> 
> @parent_ip - This is the instruction pointer of the function that called the
>       the function being traced (where the call of the function occurred).
> 
> @op - This is a pointer to ftrace_ops that was used to register the callback.
>       This can be used to pass data to the callback via the private pointer.
> 
> @regs - If the FTRACE_OPS_FL_SAVE_REGS or FTRACE_OPS_FL_SAVE_REGS_IF_SUPPORTED
>       flags are set in the ftrace_ops structure, then this will be pointing
>       to the pt_regs structure like it would be if an breakpoint was placed
>       at the start of the function where ftrace was tracing. Otherwise it
>       either contains garbage, or NULL.
> 
> 
> The ftrace FLAGS
> ================
> 
> The ftrace_ops flags are all defined and documented in include/linux/ftrace.h.
> Some of the flags are used for internal infrastructure of ftrace, but the
> ones that users should be aware of are the following:
> 
> (All of these are prefixed with FTRACE_OPS_FL_)

I hate to say it, but you'll reduce the cognitive load on the reader if you
just spell the flags out.

> PER_CPU - When set, the callback can be enabled or disabled per cpu with the
>       following functions:
> 
>       void ftrace_function_local_enable(struct ftrace_ops *ops);
>       void ftrace_function_local_disable(struct ftrace_ops *ops);
> 
>       These two functions must be called with preemption disabled.

s/per cpu/per-CPU/

More importantly, though, what does this actually mean?  Presumably that
the callback is only called if the function is hit on an enabled CPU?  It
seems you can only enable/disable the local CPU?  Is the default state
enabled or disabled?  Inquiring minds want to know.

> SAVE_REGS - If the callback requires reading or modifying the pt_regs
>       passed to the callback, then it must set this flag. Registering
>       a ftrace_ops with this flag set on an architecture that does not
>       support passing of pt_regs to the callback, will fail.

Drop that last comma :)

> SAVE_REGS_IF_SUPPORTED - Similar to SAVE_REGS but the registering of a
>       ftrace_ops on an architecture that does not support passing of regs
>       will not fail with this flag set. But the callback must check if
>       regs is NULL or not to determine if the architecture supports it.
> 
> RECURSION_SAFE - By default, a wrapper is added around the callback to
>       make sure that recursion of the function does not occur. That is
>       if a function within the callback itself is also traced, ftrace

s/within the/called by the/

>       will prevent the callback from being called again. But this wrapper
>       adds some overhead, and if the callback is safe from recursion,
>       it can set this flag to disable the ftrace protection.

What happens if you mark a function safe and it recurses anyway?  Say, if
somebody else has hooked a different function unknown to the current
caller? 

> IPMODIFY - Requires SAVE_REGS set. If the callback is to "hijack" the
>       traced function (have another function called instead of the traced
>       function), it requires setting this flag. This is what live kernel
>       patches uses. Without this flag the pt_regs->ip can not be modified.
>       Note, only one ftrace_ops with IPMODIFY set may be registered to
>       any given function at a time.

I assume this requires being able to get the regs too?  

> RCU - If this is set, then the callback will only be called by functions
>       where RCU is "watching". This is required if the callback function
>       performs any rcu_read_lock() operation.

What does that mean, exactly?  When might RCU not be watching?


> Filtering what functions to trace

s/what/which/

> =================================
> 
> If a callback is only to be called from specific functions, a filter must be
> set up. The filters are added by name, or ip if it is known.
> 
>  int ftrace_set_filter(struct ftrace_ops *ops, unsigned char *buf,
> 		       int len, int reset);
> 
>  @ops - the ops to set the filter with
>  @buf - the string that holds the function filter text.
>  @len - the length of the string.
>  @reset - non zero to reset all filters before applying this filter.
> 
>  Filters denote which functions should be enabled when tracing is enabled.
>  If @buf is NULL and reset is set, all functions will be enabled for tracing.

...maybe a pointer to documentation on the full filter-string syntax?

> The @buf can also be a glob expression to enable all functions that
> match a specific pattern.
> 
> To just trace the schedule function:
> 
>  ret = ftrace_set_filter(&ops, "schedule", strlen("schedule"), 0);
> 
> To add more functions, call the ftrace_set_filter() more than once with the
> @reset parameter set to zero. To remove the current filter and replace it
> with new functions to trace, have @reset be non zero.

non-zero

> Sometimes more than one function has the same name. To trace just a specific
> function in this case, ftrace_set_filter_ip() can be used.
> 
>  ret = ftrace_set_filter_ip(&ops, ip, 0, 0);
> 
> Although the ip must be the address where the call to fentry or mcount is
> located in the function.

How do I get that if I know I want to trace foo() in bar.c?

> If a glob is used to set the filter, to remove unwanted matches the
> ftrace_set_notrace() can also be used.
> 
>   int ftrace_set_notrace(struct ftrace_ops *ops, unsigned char *buf,
> 			 int len, int reset);
> 
> This takes the same parameters as ftrace_set_filter() but will add the
> functions it finds to not be traced. This doesn't remove them from the
> filter itself, but keeps them from being traced. If @reset is set,
> the filter is cleaded but the functions that match @buf will still not

cleaded? :)

> be traced (the callback will not be called on those functions).

So how do you clead the "notrace" list?

jon