Re: [RFC][PATCH 00/14] function_graph: Rewrite to allow multiple users

[Masami Hiramatsu]

Hi Steve,

On Wed, 21 Nov 2018 20:27:08 -0500
Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:

> 
> I talked with many of you at Plumbers about rewriting the function graph
> tracer. Well, this is it. I was originally going to produce just a
> proof of concept, but when I found that I had to fix a design flaw
> and that covered all the arch code anyway, I decided to do more of a
> RFC patch set.

Thank you for starting this work! This might be a good way to simplify
treating of shadow stacks in kretprobe and function-graph-tracers.
(And I hope this can help me to remove some kind of complexity in
 kretprobes)

> 
> I probably should add more comments to the code, and update the 
> function graph design documentation, but I wanted to get this out
> before the US Turkey day for your enjoyment while you try to let your
> pants buckle again.

:)

Let me try to review and port kretprobe on it. On the way, I will find
issues if there are.

> 
> Why the rewrite? 
> 
> Well the fuction graph tracer is arguably the strongest of the tracers.
> It shows both the entrance and exit of a function, can give the timings
> of a function, and shows the execution of the code quite nicely.
> 
> But it has one major flaw.
> 
> It can't let more than one user access it at a time. The function
> tracer has had that feature for years now, but due to the design of
> the function graph tracer it was difficult to implement. Why?
> 
> Because you must maintain the state of a three-tuple.
> 
>  Task, Function, Callback
> 
> The state is determined at by the entryfunc and must be passed to the
> retfunc when the function being traced returns. But this is not an
> easy task, as that state can be different for each task, each function
> and each callback.
> 
> What's the solution? I use the shadow stack that is already being
> used to store the function return addresses.
> 
>   A big thanks to Masami Hiramatsu for suggesting this idea!
> 
> For now, I only allow an 16 users of the function graph tracer at a time.
> That should be more than enough. I create an array of 16 fgraph_ops
> pointers. When a user registers their fgraph_ops to the function graph
> tracer, it is assigned an index into that array, which will hold a pointer
> to the fgraph_ops being registered.
> 
> On entry of the function, the array is iterated and each entryfunc of
> the fgraph_ops in the array is called. If the entryfunc returns non-zero,
> then the index of that fgraph_ops is pushed on the shadow stack (along
> with the index to the "ret_stack entry" structure, for fast access
> to it). If the entryfunc returns zero, then it is ignored. If at least
> one function returned non-zero then the return of the traced function
> will also be traced.
> 
> On the return of the function, the shadow stack is examined and all
> the indexes that were pushed on the stack is read, and each fgraph_ops
> retfunc is called in the reverse order.
> 
> When a fgraph_ops is unregistered, its index in the array is set to point
> to a "stub" fgraph_ops that holds stub functions that just return
> "0" for the entryfunc and does nothing for the retfunc. This is because
> the retfunc may be called literally days after the entryfunc is called
> and we want to be able to free the fgraph_ops that is unregistered.
> 
> Note, if another fgraph_ops is registered in the same location, its
> retfunc may be called that was set by a previous fgraph_ops. This
> is not a regression because that's what can happen today if you unregister
> a callback from the current function_graph tracer and register another
> one. If this is an issue, there are ways to solve it.

Yeah, I need the solution, maybe an API to get correct return address? :)

By the way, are there any way to hold a private data on each ret_stack entry?
Since kretprobe supports "entry data" passed from entry_handler to
return handler, we have to store the data or data-instance on the ret_stack.

This feature is used by systemtap to save the function entry data, like
function parameters etc., so that return handler analyzes the parameters
with return value.

Thank you,

-- 
Masami Hiramatsu <mhiramat@xxxxxxxxxx>