Re: tracepoint maintainance models

From: Nicholas Miell
Date: Sun Sep 17 2006 - 16:29:02 EST


On Sun, 2006-09-17 at 16:36 +0200, Ingo Molnar wrote:
> * Paul Mundt <lethal@xxxxxxxxxxxx> wrote:
>
> > What exactly are you trying to prove with this? Yes, people aren't
> > opposed to a lightweight marker facility. Ingo made some suggestions
> > regarding that, and others (Andrew, Martin, etc.) have pointed out
> > that this would also be beneficial for certain use cases. I don't see
> > anyone violently opposed to lightweight markers, I see people
> > violently opposed to the ltt-centric breed of static instrumentation
> > (and yes, I'm one of them), let's not confuse the two.
>
> yes. The way i see this whole issue (and what i've been trying argue for
> a long time) is that with dynamic tracers we have a _continuum_ of
> _tracepoint maintainance models_ that maintainers can choose from, each
> of which model gives the same "end-user experience":

To inject some facts into this argument, I took a look at dtrace on a
Solaris LiveCD (Belenix 0.4.4, actually, and wow are their userspace
apps are as terrible as I've been lead to be believe.)

On my system, Solaris has 49 "real" static probes (with actual
documentation[1]). They are as follows:

io:::done proc:::lwp-start
io:::start proc:::signal-clear
io:::wait-done proc:::signal-discard
io:::wait-start proc:::signal-handle
lockstat:::adaptive-acquire proc:::signal-send
lockstat:::adaptive-block proc:::start
lockstat:::adaptive-release sched:::change-pri
lockstat:::adaptive-spin sched:::dequeue
lockstat:::rw-acquire sched:::enqueue
lockstat:::rw-block sched:::off-cpu
lockstat:::rw-downgrade sched:::on-cpu
lockstat:::rw-release sched:::preempt
lockstat:::rw-upgrade sched:::remain-cpu
lockstat:::spin-acquire sched:::schedctl-nopreempt
lockstat:::spin-release sched:::schedctl-preempt
lockstat:::spin-spin sched:::schedctl-yield
lockstat:::thread-spin sched:::sleep
proc:::create sched:::surrender
proc:::exec sched:::tick
proc:::exec-failure sched:::wakeup
proc:::exec-success sdt:::callout-end
proc:::exit sdt:::callout-start
proc:::fault sdt:::interrupt-complete
proc:::lwp-create sdt:::interrupt-start
proc:::lwp-exit

You'll note that these probes are all generic high-level concepts, some
of which occur at multiple places within the kernel (You can just trust
me on this, the dtrace -l output lists multiple function sites for the
provider:::name pair).

In addition to those 49 probes, there are 330 more documented probes
which fire whenever a statistical counter changes (most of them are SNMP
MIB counters, but there are also probes related to VM behavior,
filesystem activity, etc.). These are all hidden inside the pre-existing
counter update macros[2] and didn't increase the kernel maintenance
burden because the counters already had to be maintained (which is why I
don't consider them "real").

There are also 134 more undocumented driver-specific probes. Every probe
comes labeled with a stability indicator that looks like something like
this:

8271 sdt zfs arc_evict_ghost arc-delete

Probe Description Attributes
Identifier Names: Private
Data Semantics: Private
Dependency Class: Unknown

Argument Attributes
Identifier Names: Private
Data Semantics: Private
Dependency Class: ISA

Argument Types
None

Which basically says that this undocumented probe is for private Sun use
and if you touch it and something breaks, you were warned and it's your
own damn fault[3]. (Obviously, the stable probes have different
labeling.) Also, given a D script, the dtrace command can spit out a
summary of that script's stability based on the probes it uses, which is
handy for judging the future compatibility of a script.

So, Solaris has a grand total of 513 statically defined probe points,
most of them hidden inside macros that were already there.

Then why is dtrace useful?

Because there's 48288 dynamically defined probes on function entry and
exit and another 454 dynamic syscall entry and exit probes.

This is the important part: In a dynamic tracing system, the number of
static probes necessary for the tracing system to be useful is
drastically, dramatically, absurdly lower than in a purely static
tracing system. Hell, you don't even need the static probes for it to be
useful, they're just a convenience for events which happen in multiple
places or a high-level name for a low-level implementation detail.

In order for the static tracing system to be as useful as the dynamic
system, all of those dynamically generated probe points would have to be
manually added to the kernel. The maintenance burden of this number of
probes is stupidly high. In reality, no static system would ever reach
that level of coverage.




[1] http://docs.sun.com/app/docs/doc/817-6223
[2] http://blogs.sun.com/tpenta/entry/dtrace_using_placing_sdt_probes
[3] http://docs.sun.com/app/docs/doc/817-6223/6mlkidlnp?a=view

--
Nicholas Miell <nmiell@xxxxxxxxxxx>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/