Re: Ftrace vs perf user page fault statistics differences

[Steven Rostedt]

On Mon, 19 Jun 2017 13:13:27 -0400
Will Hawkins <whh8b@xxxxxxxxxxxx> wrote:

> On Wed, Jun 14, 2017 at 4:01 PM, Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
> > On Wed, 14 Jun 2017 13:47:17 -0400
> > Will Hawkins <whh8b@xxxxxxxxxxxx> wrote:
> >
> >  
> >> > That's how trace-cmd parses it.  
> >>
> >> In the kernel version that I am running (again, pretty old) I do not
> >> have this file. I do, however, have  
> >
> > It may be due to the kernel version. It gets the functions
> > from /proc/kallsyms. That could have have an issue. Although, I have
> > this too:
> >
> > # grep per_cpu_start /proc/kallsyms
> > 0000000000000000 A __per_cpu_start  
> 
> Here's my result of a similar command:
> 
> # cat /proc/kallsyms | grep per_cpu_start
> 0000000000000000 D __per_cpu_start

Right, that it doesn't have the latest code to say to ignore it in your
kernel.

> 
> There's only the difference between (what I think is) the flag value
> there in the second column.
> 
> >
> > But I don't see it being converted in my report.
> >
> > Hmm, it's not saved. Interesting:
> >
> >  trace-cmd report -f
> >
> > to see the list of saved functions. I need to figure out why it does
> > for you, but not for me.  
> 
> sudo ./trace-cmd report -f | grep per_cpu_start
> 0000000000000000 __per_cpu_start

Yep, it took it. It only saves the address an the name.

> > Ah, I bet it's a change in the kernel. A recent update to trace-cmd was
> > to ignore functions in kallsyms of type "A" (which you can see is what
> > I have above).  
> 
> And my output (above) seems to show that mine shows the per_cpu_start
> as a D instead of an A. Perhaps that is why we are seeing differences
> in our report? It's still curious that it would match 0x000..0 with
> something that is clearly, well, not 0x00...0.

Because function ips are seldom at the start of the function (well,
other than fentry code), it grabs the first function with the address
before the ip. Which would be 0x0000.

> 
> Let me know if there is a spot in the code where it decides whether to
> symbolize. Or, I can work backward from the recent change you
> mentioned above to find that spot if you will tell me the hash of that
> commit.

If you run with trace-cmd report -O offset, you should see the offset
off of functions. Not sure it works with function graph tracer, but
should at least work with function tracing.

-- Steve