Re: Re: [PATCH 14/28] ktap: add runtime/kp_events.[c|h]

From: Masami Hiramatsu
Date: Tue Apr 01 2014 - 03:00:21 EST

(2014/03/31 19:14), Jovi Zhangwei wrote:
> On Mon, Mar 31, 2014 at 5:10 PM, Masami Hiramatsu
> <> wrote:
>> (2014/03/28 22:47), Jovi Zhangwei wrote:
>>> kp_events.c handle ktap events management(registry, destroy, event callback)
>>> This file is core event management interface between ktap and kernel.
>>> Exposed functions:
>>> 1). kp_events_init/kp_events_exit
>>> 2). kp_event_create_kprobe
>>> create kprobe event, for example:
>>> kdebug.kprobe("SyS_futex", function () {})
>>> 3). kp_event_create_tracepoint
>>> create tracepoint event, for example"
>>> kdebug.tracepoint("sys_futex_enter", function () {})
>>> 4). kp_event_create
>>> create perf backend event, for example:
>>> trace sched:sched_switch { print(argstr) }
>>> It call kernel function 'perf_event_create_kernel_counter' to
>>> register event(tracepoint/kprobe/uprobe)
>>> 5). kp_event_getarg
>>> get argument of event, from arg0 to arg9,
>>> only can be called in probe context.
>>> trace sched:sched_switch { print(arg0, arg1) }
>>> 6). kp_event_stringify/kp_event_tostr
>>> stringify argstr, sometimes if store argstr as key to table,
>>> then it need to stringify firstly, like below:
>>> var s={} trace sched:sched_switch { s[argstr] += 1 }
>>> (This is quite rare usage, but ktap support it)
>>> Note:
>>> Why ktap support 'kdebug.kprobe' and 'kdebug.tracepoint' when
>>> it already support perf backend event(trace xxx {})?
>>> Because benchmark shows raw kprobe and tracpoint interface is faster
>>> than perf backed tracing, nearly 10+%, it's more fair to compare
>>> with Systemtap by raw tracing syntax, not perf backend tracing.
>> Do we really need it just for a +10% performance? I doubt that.
>> I think the benefit point of ktap is "dynamic & simple programmable
>> tracer in kernel", not the good performance at least at this point.
>> Thus I think we should start ktap only with perf backend.
> Yeah, agreed, most people like the perf-backed tracing syntax,
> that raw trace interface is just for benchmark when I wanted to look
> overhead compare with stap, the result is very inspiring, ktap table
> operation overhead is lower than stap.
> On the performance overhead of dynamic tracing tools(ktap/stap/dtrace),
> it's interesting enough that dtrace was used in production many year,
> _but_ IMO the runtime of dtrace is slow after I checked dtrace source
> code :), system workload does big matter than tracing tool overhead.

Yeah, I see that less overhead is also required especially for enterprise
people. I just doubt that it is solved by ktap itself. Should we improve
perf(or ftrace) to export more effective interfaces for this kind of

Thank you,

IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at