[PATCH v5 00/10] tracing: 'hist' triggers

From: Tom Zanussi
Date: Wed May 20 2015 - 17:23:12 EST

This is v5 of the 'hist triggers' patchset, following feedback from
v4.

Changes from v4:

This version addresses some problems and suggestions made by Daniel
Wagner - a lot of the code was reworked to get rid of the distinction
between keys and values, and as a result, both keys and values can be
used as sort keys. As suggested, it also allows 'val=' to be absent
in a trigger command - if no 'val' is specified, hitcount is assumed
and automatically used as the only val.

The map code was also separated out into a separate file,
tracing_map.c, allowing it to be reused. It also adds a second tracer
called function_hist that actually does reuse the code, as an RFC
patch.

Patch 01/10 [tracing: Update cond flag when enabling or disabling..]
is a fix for a problem noticed by Daniel and that fixes a problem in
existing trigger code and should be applied regardless of whether the
rest of the patchset is merged.

As mentioned, patch 10/10 is an RFC patch implementing a new tracer
based on the function tracer code. It's a fun little tool and is
useful for a specific problem I'm working on (and is also a nice test
of the tracing_map code), but is an RFC because first, I'm not sure it
would really be of general interest and secondly, it's POC-level
quality and I'd need to spend more time fixing it up to make it
upstreamable, but I don't want to waste my time if not.

There are a couple of important bits of functionality that were
present in v1 but not yet reimplemented in v5.

The first is support for compound keys. Currently, maps can only be
keyed on a single event field, whereas in v1 they could be keyed on
multiple keys. With support for compound keys, you can create much
more interesting output, such as for example per-pid lists of
syscalls or read counts e.g.:

# echo 'hist:keys=common_pid.execname,id.syscall:vals=hitcount' > \
/sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger

# cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/hist

key: common_pid:bash[3112], id:sys_write vals: count:69
key: common_pid:bash[3112], id:sys_rt_sigprocmask vals: count:218

key: common_pid:update-notifier[3164], id:sys_poll vals: count:37
key: common_pid:update-notifier[3164], id:sys_recvfrom vals: count:118

key: common_pid:deja-dup-monito[3194], id:sys_sendto vals: count:1
key: common_pid:deja-dup-monito[3194], id:sys_read vals: count:4
key: common_pid:deja-dup-monito[3194], id:sys_poll vals: count:8
key: common_pid:deja-dup-monito[3194], id:sys_recvmsg vals: count:8
key: common_pid:deja-dup-monito[3194], id:sys_getegid vals: count:8

key: common_pid:emacs[3275], id:sys_fsync vals: count:1
key: common_pid:emacs[3275], id:sys_open vals: count:1
key: common_pid:emacs[3275], id:sys_symlink vals: count:2
key: common_pid:emacs[3275], id:sys_poll vals: count:23
key: common_pid:emacs[3275], id:sys_select vals: count:23
key: common_pid:emacs[3275], id:unknown_syscall vals: count:34
key: common_pid:emacs[3275], id:sys_ioctl vals: count:60
key: common_pid:emacs[3275], id:sys_rt_sigprocmask vals: count:116

key: common_pid:cat[3323], id:sys_munmap vals: count:1
key: common_pid:cat[3323], id:sys_fadvise64 vals: count:1

Related to that is support for sorting on multiple fields. Currently,
you can sort using only a primary key. Being able to sort on multiple
or at least a secondary key is indispensible for seeing trends when
displaying multiple values.

Changes from v3:

v4 fixes the race in tracing_map_insert() noted in v3, where
map.val.key could be checked even if map.val wasn't yet set. The
simple fix for that in tracing_map_insert() introduces the possibility
of duplicates in the map, which though rare, need to be accounted for
in the output. To address that, duplicate-merging code was added to
the map-printing code.

It was also pointed out that it didn't seem correct to include
module.h, but the fix for that has deeper roots and is being addressed
by a separate patchset; for now we need to continue including
module.h, though prompted by that I did some other header include
cleanup.

The functionality remains the same as v2, but this version no longer
tries to export and use bpf_maps, and more importantly removes the
associated GFP_NOTRACE/trace event hacks and kmem macros required to
work around the bpf_map implementation.

The tracing_map functionality is instead built on top of a simple
lock-free map algorithm originated by Dr. Cliff Click (see references
in the code for more details), which though too restrictive to be
general-purpose in its current form, functions nicely as a
special-purpose tracing map.

v3 also moves the hist triggers code into a separate file and puts it
all behind a new config option, CONFIG_HIST_TRIGGERS. It also merges
in the sorting code rather than keeping it as a separate patch.

This patchset also includes a couple other new and related triggers,
enable_hist and disable_hist, very similar to the existing
enable_event/disable_event triggers used to automatically enable and
disable events based on a triggering condition, but in this case
allowing hist triggers to be enabled and disabled in the same way.

- Added an insert check for val before checking the key associated with val
- Added code to merge possible duplicates in the map

Changes from v2:
- reimplemented tracing_map, replacing bpf_map with nmi-safe/lock-free map
- removed GPF_NOTRACE, kmalloc/free macros and event hacks needed by bpf_maps
- moved hist triggers from trace_events_trigger.c to trace_events_hist.c
- added CONFIG_HIST_TRIGGERS config option
- consolidated sorting code with main patch

Changes from v1:
- completely rewritten on top of tracing_map (renamed and exported bpf_map)
- added map clearing and client ops to tracing_map
- changed the name from 'hash' triggers to 'hist' triggers
- added new trigger 'pause' feature
- added new enable_hist and disable_hist triggers
- added usage for hist/enable_hist/disable hist to tracing/README
- moved examples into Documentation/trace/event.txt
- added ___GFP_NOTRACE, kmalloc/kfree macros, and conditional kmem tracepoints

The following changes since commit f267caab44451baa70d25fa3191a68bb79ad1b08:

tracing: Remove unused prototype ftrace_event_define_field() (2015-05-13 14:03:51 -0400)

are available in the git repository at:

git://git.yoctoproject.org/linux-yocto-contrib.git tzanussi/hist-triggers-v5
http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-contrib/log/?h=tzanussi/hist-triggers-v5

Tom Zanussi (10):
tracing: Update cond flag when enabling or disabling a trigger
tracing: Make ftrace_event_field checking functions available
tracing: Add event record param to trigger_ops.func()
tracing: Add get_syscall_name()
tracing: Add a per-event-trigger 'paused' field
trace: Add lock-free tracing_map
tracing: Add 'hist' event trigger command
tracing: Add enable_hist/disable_hist triggers
tracing: Add 'hist' trigger Documentation
ftrace: Add function_hist tracer

Documentation/trace/events.txt | 872 ++++++++++++++++++++++++
include/linux/ftrace_event.h | 9 +-
kernel/trace/Kconfig | 14 +
kernel/trace/Makefile | 3 +
kernel/trace/trace.c | 58 ++
kernel/trace/trace.h | 88 ++-
kernel/trace/trace_events.c | 4 +
kernel/trace/trace_events_filter.c | 12 -
kernel/trace/trace_events_hist.c | 1238 +++++++++++++++++++++++++++++++++++
kernel/trace/trace_events_trigger.c | 149 +++--
kernel/trace/trace_functions_hist.c | 345 ++++++++++
kernel/trace/trace_syscalls.c | 11 +
kernel/trace/tracing_map.c | 831 +++++++++++++++++++++++
kernel/trace/tracing_map.h | 133 ++++
14 files changed, 3683 insertions(+), 84 deletions(-)
create mode 100644 kernel/trace/trace_events_hist.c
create mode 100644 kernel/trace/trace_functions_hist.c
create mode 100644 kernel/trace/tracing_map.c
create mode 100644 kernel/trace/tracing_map.h

--
1.9.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/