[PATCH 12/15] tracing: Add 'hist' event trigger command

From: Tom Zanussi
Date: Mon Mar 02 2015 - 11:05:20 EST

'hist' triggers allow users to continually aggregate trace events,
which can then be viewed afterwards by simply reading a 'hist' file
containing the aggregation in a human-readable format.

The basic idea is very simple and boils down to a mechanism whereby
trace events, rather than being exhaustively dumped in raw form and
viewed directly, are automatically 'compressed' into meaningful tables
completely defined by the user.

This is done strictly via single-line command-line commands and
without the aid of any kind of programming language or interpreter.

A surprising number of typical use cases can be accomplished by users
via this simple mechanism. In fact, a large number of the tasks that
users typically do using the more complicated script-based tracing
tools, at least during the initial stages of an investigation, can be
accomplished by simply specifying a set of keys and values to be used
in the creation of a hash table.

The Linux kernel trace event subsystem happens to provide an extensive
list of keys and values ready-made for such a purpose in the form of
the event format files associated with each trace event. By simply
consulting the format file for field names of interest and by plugging
them into the hist trigger command, users can create an endless number
of useful aggregations to help with investigating various properties
of the system. See Documentation/trace/events.txt for examples.

hist triggers are implemented on top of the existing event trigger
infrastructure, and as such are consistent with the existing triggers
from a user's perspective as well.

The basic syntax follows the existing trigger syntax. Users start an
aggregation by writing a 'hist' trigger to the event of interest's
trigger file:

# echo hist:keys=xxx:values=yyy [ if filter] > event/trigger

Once a hist trigger has been set up, by default it continually
aggregates every matching event into a hash table using the event key
and value fields specified.

To view the aggregation at any point in time, simply read the 'hist'
file in the same directory as the 'trigger' file:

# cat event/hist

The detailed syntax provides additional options for user control, and
is described exhaustively in Documentation/trace/events.txt and in the
virtual tracing/README file in the tracing subsystem.

Signed-off-by: Tom Zanussi <tom.zanussi@xxxxxxxxxxxxxxx>
---
include/linux/ftrace_event.h | 1 +
kernel/trace/trace.c | 39 ++
kernel/trace/trace.h | 1 +
kernel/trace/trace_events.c | 3 +
kernel/trace/trace_events_filter.c | 3 +-
kernel/trace/trace_events_trigger.c | 1014 +++++++++++++++++++++++++++++++++++
6 files changed, 1060 insertions(+), 1 deletion(-)

diff --git a/include/linux/ftrace_event.h b/include/linux/ftrace_event.h
index c967526..483e011 100644
--- a/include/linux/ftrace_event.h
+++ b/include/linux/ftrace_event.h
@@ -416,6 +416,7 @@ enum event_trigger_type {
ETT_SNAPSHOT = (1 << 1),
ETT_STACKTRACE = (1 << 2),
ETT_EVENT_ENABLE = (1 << 3),
+ ETT_EVENT_HIST = (1 << 4),
};

extern int filter_match_preds(struct event_filter *filter, void *rec);
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 60fa9e1..683048f 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -3730,6 +3730,7 @@ static const char readme_msg[] =
#ifdef CONFIG_TRACER_SNAPSHOT
"\t\t snapshot\n"
#endif
+ "\t\t hist (see below)\n"
"\t example: echo traceoff > events/block/block_unplug/trigger\n"
"\t echo traceoff:3 > events/block/block_unplug/trigger\n"
"\t echo 'enable_event:kmem:kmalloc:3 if nr_rq > 1' > \\\n"
@@ -3745,6 +3746,44 @@ static const char readme_msg[] =
"\t To remove a trigger with a count:\n"
"\t echo '!<trigger>:0 > <system>/<event>/trigger\n"
"\t Filters can be ignored when removing a trigger.\n"
+ " hist trigger\t- If set, event hits are aggregated into a hash table\n"
+ "\t Format: hist:keys=<field1>:values=<field1[,field2,...]>\n"
+ "\t [:size=#entries][:sort=field1][:pause][:continue]\n"
+ "\t [:clear] [if <filter>]\n\n"
+ "\t When a matching event is hit, an entry is added to a hash\n"
+ "\t table using the key(s) and value(s) named. Keys and values\n"
+ "\t correspond to fields in the event's format description.\n"
+ "\t Values must correspond to numeric fields - on an event hit,\n"
+ "\t the value(s) will be added to a sum kept for that field.\n"
+ "\t The special string 'hitcount' can be used in place of an\n"
+ "\t explicit value field - this is simply a count of event hits.\n"
+ "\t Keys can be any field, or the special string 'stacktrace',\n"
+ "\t which will use the event's kernel stacktrace as the key.\n\n"
+ "\t Reading the 'hist' file for the event will dump the hash\n"
+ "\t table in its entirety to stdout. By default, numeric fields\n"
+ "\t are displayed as base-10 integers. This can be modified by\n"
+ "\t appending any of the following modifiers to the field name:\n\n"
+ "\t .hex display a number as a hex value\n"
+ "\t .sym display an address as a symbol\n"
+ "\t .syscall display a syscall id as a system call name\n"
+ "\t .execname display a common_pid as a program name\n\n"
+ "\t By default, the size of the hash table is 2048 entries. The\n"
+ "\t 'size' param can be used to specify more or fewer than that.\n"
+ "\t The units are in terms of hashtable entries - if a run uses\n"
+ "\t more entries than specified, the results will show the number\n"
+ "\t of 'drops', the number of hits that were ignored. The size\n"
+ "\t should be a power of 2 between 128 and 131072 (any non-\n"
+ "\t power-of-2 number specified will be rounded up).\n\n"
+ "\t The 'sort' param can be used to specify a value field to sort\n"
+ "\t on. The default if unspecified is 'hitcount' and the.\n"
+ "\t default sort order is 'ascending'. To sort in the opposite\n"
+ "\t direction, append .descending' to the sort key.\n"
+ "\t The 'pause' param can be used to pause an existing hist\n"
+ "\t trigger or to start a hist trigger but not log any events\n"
+ "\t until told to do so. 'continue' can be used to start or\n"
+ "\t restart a paused hist trigger.\n\n"
+ "\t The 'clear' param will clear the contents of a running hist\n"
+ "\t trigger and leave its current paused/active state.\n"
;

static ssize_t
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 5bc1752..f2efd6a 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -1098,6 +1098,7 @@ extern struct mutex event_mutex;
extern struct list_head ftrace_events;

extern const struct file_operations event_trigger_fops;
+extern const struct file_operations event_hist_fops;

extern int register_trigger_cmds(void);
extern void clear_event_triggers(struct trace_array *tr);
diff --git a/kernel/trace/trace_events.c b/kernel/trace/trace_events.c
index 0d2e473..b906f40 100644
--- a/kernel/trace/trace_events.c
+++ b/kernel/trace/trace_events.c
@@ -1621,6 +1621,9 @@ event_create_dir(struct dentry *parent, struct ftrace_event_file *file)
trace_create_file("trigger", 0644, file->dir, file,
&event_trigger_fops);

+ trace_create_file("hist", 0444, file->dir, file,
+ &event_hist_fops);
+
trace_create_file("format", 0444, file->dir, call,
&ftrace_event_format_fops);

diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c
index aa8ab85..15b1ffa 100644
--- a/kernel/trace/trace_events_filter.c
+++ b/kernel/trace/trace_events_filter.c
@@ -911,7 +911,8 @@ int filter_assign_type(const char *type)
if (strstr(type, "__data_loc") && strstr(type, "char"))
return FILTER_DYN_STRING;

- if (strchr(type, '[') && strstr(type, "char"))
+ if (strchr(type, '[') &&
+ (strstr(type, "char") || strstr(type, "u8")))
return FILTER_STATIC_STRING;

return FILTER_OTHER;
diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
index 010ce30..80805f3 100644
--- a/kernel/trace/trace_events_trigger.c
+++ b/kernel/trace/trace_events_trigger.c
@@ -22,6 +22,9 @@
#include <linux/ctype.h>
#include <linux/mutex.h>
#include <linux/slab.h>
+#include <linux/stacktrace.h>
+
+#include <linux/bpf.h>

#include "trace.h"

@@ -139,6 +142,7 @@ static void *trigger_start(struct seq_file *m, loff_t *pos)
/* ->stop() is called even if ->start() fails */
mutex_lock(&event_mutex);
event_file = event_file_data(m->private);
+
if (unlikely(!event_file))
return ERR_PTR(-ENODEV);

@@ -255,6 +259,7 @@ static ssize_t event_trigger_regex_write(struct file *file,

mutex_lock(&event_mutex);
event_file = event_file_data(file);
+
if (unlikely(!event_file)) {
mutex_unlock(&event_mutex);
free_page((unsigned long)buf);
@@ -1431,12 +1436,1021 @@ static __init int register_trigger_traceon_traceoff_cmds(void)
return ret;
}

+struct hist_field;
+
+typedef u64 (*hist_field_fn_t) (struct hist_field *field, void *event);
+
+struct hist_field {
+ struct ftrace_event_field *field;
+ unsigned long flags;
+ hist_field_fn_t fn;
+};
+
+static u64 hist_field_none(struct hist_field *field, void *event)
+{
+ return 0;
+}
+
+static u64 hist_field_string(struct hist_field *hist_field, void *event)
+{
+ char *addr = (char *)(event + hist_field->field->offset);
+
+ return (u64)addr;
+}
+
+#define DEFINE_HIST_FIELD_FN(type) \
+static u64 hist_field_##type(struct hist_field *hist_field, void *event)\
+{ \
+ type *addr = (type *)(event + hist_field->field->offset); \
+ \
+ return (u64)*addr; \
+}
+
+DEFINE_HIST_FIELD_FN(s64);
+DEFINE_HIST_FIELD_FN(u64);
+DEFINE_HIST_FIELD_FN(s32);
+DEFINE_HIST_FIELD_FN(u32);
+DEFINE_HIST_FIELD_FN(s16);
+DEFINE_HIST_FIELD_FN(u16);
+DEFINE_HIST_FIELD_FN(s8);
+DEFINE_HIST_FIELD_FN(u8);
+
+#define HIST_TRIGGER_BITS 11
+#define HIST_TRIGGER_BITS_MAX 17
+#define HIST_TRIGGER_BITS_MIN 7
+#define HIST_VALS_MAX 8
+
+#define HIST_KEY_STRING_MAX 64
+
+enum hist_field_flags {
+ HIST_FIELD_SYM = 1,
+ HIST_FIELD_HEX = 2,
+ HIST_FIELD_STACKTRACE = 4,
+ HIST_FIELD_STRING = 8,
+ HIST_FIELD_EXECNAME = 16,
+ HIST_FIELD_SYSCALL = 32,
+};
+
+struct hist_trigger_attrs {
+ char *keys_str;
+ char *vals_str;
+ bool pause;
+ bool cont;
+ bool clear;
+ unsigned int map_bits;
+};
+
+struct hist_trigger_data {
+ atomic_t next_entry;
+ atomic64_t total_hits;
+ struct hist_field *key;
+ struct hist_field *vals[HIST_VALS_MAX];
+ unsigned int n_vals;
+ struct ftrace_event_file *event_file;
+ atomic64_t drops;
+ struct hist_trigger_attrs *attrs;
+ unsigned int map_bits;
+ struct bpf_map *map;
+ union bpf_attr map_attr;
+};
+
+struct hist_trigger_entry {
+ struct hist_trigger_data *hist_data;
+ atomic64_t hitcount;
+ atomic64_t sums[HIST_VALS_MAX];
+ char *comm;
+};
+
+#define HIST_STACKTRACE_DEPTH 16
+#define HIST_STACKTRACE_SKIP 5
+
+static hist_field_fn_t select_value_fn(int field_size, int field_is_signed)
+{
+ hist_field_fn_t fn = NULL;
+
+ switch (field_size) {
+ case 8:
+ if (field_is_signed)
+ fn = hist_field_s64;
+ else
+ fn = hist_field_u64;
+ break;
+ case 4:
+ if (field_is_signed)
+ fn = hist_field_s32;
+ else
+ fn = hist_field_u32;
+ break;
+ case 2:
+ if (field_is_signed)
+ fn = hist_field_s16;
+ else
+ fn = hist_field_u16;
+ break;
+ case 1:
+ if (field_is_signed)
+ fn = hist_field_s8;
+ else
+ fn = hist_field_u8;
+ break;
+ }
+
+ return fn;
+}
+
+static inline void save_comm(char *comm, struct task_struct *task)
+{
+ if (!task->pid) {
+ strcpy(comm, "<idle>");
+ return;
+ }
+
+ if (WARN_ON_ONCE(task->pid < 0)) {
+ strcpy(comm, "<XXX>");
+ return;
+ }
+
+ if (task->pid > PID_MAX_DEFAULT) {
+ strcpy(comm, "<...>");
+ return;
+ }
+
+ memcpy(comm, task->comm, TASK_COMM_LEN);
+}
+
+static void hist_trigger_entry_free(struct hist_trigger_entry *entry)
+{
+ kfree_notrace(entry->comm);
+ kfree_notrace(entry);
+}
+
+static struct hist_trigger_entry *
+hist_trigger_entry_create(struct hist_trigger_data *hist_data)
+{
+ struct hist_trigger_entry *entry = NULL;
+
+ if (atomic_read(&hist_data->next_entry) >=
+ hist_data->map_attr.max_entries)
+ return NULL;
+
+ atomic_inc(&hist_data->next_entry);
+
+ entry = kmalloc_notrace(sizeof(*entry), GFP_ATOMIC);
+ if (!entry)
+ return entry;
+ memset(entry, 0, sizeof(*entry));
+
+ if (hist_data->key->flags & HIST_FIELD_EXECNAME) {
+ entry->comm = kmalloc_notrace(TASK_COMM_LEN + 1, GFP_ATOMIC);
+ if (!entry->comm) {
+ hist_trigger_entry_free(entry);
+ return NULL;
+ }
+ save_comm(entry->comm, current);
+ }
+
+ entry->hist_data = hist_data;
+
+ return entry;
+}
+
+static void destroy_hist_field(struct hist_field *hist_field)
+{
+ kfree(hist_field);
+}
+
+static struct hist_field *create_hist_field(struct ftrace_event_field *field,
+ unsigned long flags)
+{
+ hist_field_fn_t fn = hist_field_none;
+ struct hist_field *hist_field;
+
+ hist_field = kzalloc(sizeof(struct hist_field), GFP_KERNEL);
+ if (!hist_field)
+ return NULL;
+
+ if (flags & HIST_FIELD_STACKTRACE) {
+ hist_field->flags = flags;
+ goto out;
+ }
+
+ if (is_function_field(field))
+ goto free;
+
+ if (is_string_field(field)) {
+ flags |= HIST_FIELD_STRING;
+ fn = hist_field_string;
+ } else {
+ fn = select_value_fn(field->size, field->is_signed);
+ if (!fn)
+ goto free;
+ }
+
+ hist_field->field = field;
+ hist_field->flags = flags;
+ hist_field->fn = fn;
+ out:
+ return hist_field;
+ free:
+ kfree(hist_field);
+ hist_field = NULL;
+
+ goto out;
+}
+
+static void destroy_hist_fields(struct hist_trigger_data *hist_data)
+{
+ unsigned int i;
+
+ destroy_hist_field(hist_data->key);
+ hist_data->key = NULL;
+
+ for (i = 0; i < hist_data->n_vals; i++) {
+ destroy_hist_field(hist_data->vals[i]);
+ hist_data->vals[i] = NULL;
+ }
+}
+
+static int create_key_field(struct hist_trigger_data *hist_data,
+ struct ftrace_event_file *file,
+ char *field_str)
+{
+ struct ftrace_event_field *field = NULL;
+ unsigned long flags = 0;
+ char *field_name;
+ int ret = 0;
+
+ if (!strcmp(field_str, "stacktrace"))
+ flags |= HIST_FIELD_STACKTRACE;
+ else {
+ field_name = strsep(&field_str, ".");
+ if (field_str) {
+ if (!strcmp(field_str, "sym"))
+ flags |= HIST_FIELD_SYM;
+ else if (!strcmp(field_str, "hex"))
+ flags |= HIST_FIELD_HEX;
+ else if (!strcmp(field_str, "execname") &&
+ !strcmp(field_name, "common_pid"))
+ flags |= HIST_FIELD_EXECNAME;
+ else if (!strcmp(field_str, "syscall"))
+ flags |= HIST_FIELD_SYSCALL;
+ }
+
+ field = trace_find_event_field(file->event_call, field_name);
+ if (!field) {
+ ret = -EINVAL;
+ goto out;
+ }
+ }
+
+ hist_data->key = create_hist_field(field, flags);
+ if (!hist_data->key)
+ ret = -ENOMEM;
+ out:
+ return ret;
+}
+
+static int create_val_field(struct hist_trigger_data *hist_data,
+ unsigned int val,
+ struct ftrace_event_file *file,
+ char *field_str)
+{
+ struct ftrace_event_field *field = NULL;
+ unsigned long flags = 0;
+ char *field_name;
+ int ret = 0;
+
+ if (!strcmp(field_str, "hitcount"))
+ return ret;
+
+ field_name = strsep(&field_str, ".");
+ if (field_str) {
+ if (!strcmp(field_str, "sym"))
+ flags |= HIST_FIELD_SYM;
+ else if (!strcmp(field_str, "hex"))
+ flags |= HIST_FIELD_HEX;
+ }
+
+ field = trace_find_event_field(file->event_call, field_name);
+ if (!field) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ hist_data->vals[val] = create_hist_field(field, flags);
+ if (!hist_data->vals[val]) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ hist_data->n_vals++;
+ out:
+ return ret;
+}
+
+static int create_hist_fields(struct hist_trigger_data *hist_data,
+ struct ftrace_event_file *file)
+{
+ char *fields_str, *field_str;
+ unsigned int i;
+ int ret;
+
+ fields_str = hist_data->attrs->keys_str;
+ strsep(&fields_str, "=");
+ if (!fields_str) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ ret = create_key_field(hist_data, file, fields_str);
+ if (ret)
+ goto out;
+
+ fields_str = hist_data->attrs->vals_str;
+ strsep(&fields_str, "=");
+ if (!fields_str) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ for (i = 0; i < HIST_VALS_MAX; i++) {
+ field_str = strsep(&fields_str, ",");
+ if (!field_str) {
+ if (i == 0) {
+ ret = -EINVAL;
+ goto out;
+ } else
+ break;
+ }
+ ret = create_val_field(hist_data, i, file, field_str);
+ if (ret)
+ goto out;
+ }
+ out:
+ return ret;
+}
+
+static u32 get_key_size(struct hist_trigger_data *hist_data)
+{
+ u32 key_size;
+
+ if (hist_data->key->flags & HIST_FIELD_STACKTRACE)
+ key_size = sizeof(unsigned long) * HIST_STACKTRACE_DEPTH;
+ else
+ key_size = hist_data->key->field->size;
+
+ return key_size;
+}
+
+static void destroy_hist_trigger_attrs(struct hist_trigger_attrs *attrs)
+{
+ kfree(attrs->keys_str);
+ kfree(attrs->vals_str);
+ kfree(attrs);
+}
+
+static int parse_map_size(char *str)
+{
+ unsigned long size, map_bits;
+ int ret;
+
+ strsep(&str, "=");
+ if (!str) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ ret = kstrtoul(str, 0, &size);
+ if (ret)
+ goto out;
+
+ map_bits = ilog2(roundup_pow_of_two(size));
+ if (map_bits < HIST_TRIGGER_BITS_MIN ||
+ map_bits > HIST_TRIGGER_BITS_MAX)
+ ret = -EINVAL;
+ else
+ ret = map_bits;
+ out:
+ return ret;
+}
+
+static struct hist_trigger_attrs *parse_hist_trigger_attrs(char *trigger_str)
+{
+ struct hist_trigger_attrs *attrs;
+ int ret = 0;
+
+ attrs = kzalloc(sizeof(*attrs), GFP_KERNEL);
+ if (!attrs)
+ return ERR_PTR(-ENOMEM);
+
+ while (trigger_str) {
+ char *str = strsep(&trigger_str, ":");
+
+ if (!strncmp(str, "keys", strlen("keys")) ||
+ !strncmp(str, "key", strlen("key")))
+ attrs->keys_str = kstrdup(str, GFP_KERNEL);
+ else if (!strncmp(str, "values", strlen("values")) ||
+ !strncmp(str, "vals", strlen("vals")) ||
+ !strncmp(str, "val", strlen("val")))
+ attrs->vals_str = kstrdup(str, GFP_KERNEL);
+ else if (!strncmp(str, "pause", strlen("pause")))
+ attrs->pause = true;
+ else if (!strncmp(str, "continue", strlen("continue")) ||
+ !strncmp(str, "cont", strlen("cont")))
+ attrs->cont = true;
+ else if (!strncmp(str, "clear", strlen("clear")))
+ attrs->clear = true;
+ else if (!strncmp(str, "size", strlen("size"))) {
+ int map_bits = parse_map_size(str);
+ if (map_bits < 0) {
+ ret = map_bits;
+ goto free;
+ }
+ attrs->map_bits = map_bits;
+ } else {
+ ret = -EINVAL;
+ goto free;
+ }
+ }
+
+ return attrs;
+ free:
+ destroy_hist_trigger_attrs(attrs);
+
+ return ERR_PTR(ret);
+}
+
+static void destroy_hist_data(struct hist_trigger_data *hist_data)
+{
+ synchronize_sched();
+
+ destroy_hist_trigger_attrs(hist_data->attrs);
+
+ destroy_hist_fields(hist_data);
+
+ if (hist_data->map)
+ tracing_map_destroy(hist_data->map);
+
+ kfree(hist_data);
+}
+
+static void hist_trigger_free_elem(void *value)
+{
+ hist_trigger_entry_free(*(struct hist_trigger_entry **)value);
+}
+
+static struct bpf_map_client_ops hist_map_client_ops = {
+ .free_elem = hist_trigger_free_elem,
+};
+
+static struct hist_trigger_data *
+create_hist_data(unsigned int map_bits,
+ struct hist_trigger_attrs *attrs,
+ struct ftrace_event_file *file)
+{
+ unsigned int hist_map_size = (1 << map_bits);
+ struct hist_trigger_data *hist_data;
+ int ret = 0;
+
+ hist_data = kzalloc(sizeof(*hist_data), GFP_KERNEL);
+ if (!hist_data)
+ return NULL;
+
+ hist_data->map_attr.map_type = BPF_MAP_TYPE_HASH;
+ hist_data->map_attr.max_entries = hist_map_size;;
+
+ hist_data->attrs = attrs;
+
+ ret = create_hist_fields(hist_data, file);
+ if (ret < 0)
+ goto free;
+
+ hist_data->map_attr.key_size = get_key_size(hist_data);
+ hist_data->map_attr.value_size = sizeof(struct hist_trigger_entry *);
+
+ hist_data->map = tracing_map_create(&hist_data->map_attr,
+ &hist_map_client_ops);
+ if (IS_ERR(hist_data->map)) {
+ ret = PTR_ERR(hist_data->map);
+ goto free;
+ }
+
+ hist_data->map_bits = map_bits;
+ hist_data->event_file = file;
+ out:
+ return hist_data;
+ free:
+ destroy_hist_data(hist_data);
+ if (ret)
+ hist_data = ERR_PTR(ret);
+ else
+ hist_data = NULL;
+
+ goto out;
+}
+
+static int hist_trigger_entry_insert(struct hist_trigger_data *hist_data,
+ void *key,
+ struct hist_trigger_entry *val)
+{
+ return tracing_map_update_elem(hist_data->map, key, &val,
+ &hist_data->map_attr);
+}
+
+static void hist_trigger_entry_update(struct hist_trigger_data *hist_data,
+ struct hist_trigger_entry *entry,
+ void *rec)
+{
+ struct hist_field *hist_field;
+ unsigned int i;
+ u64 hist_val;
+
+ for (i = 0; i < hist_data->n_vals; i++) {
+ hist_field = hist_data->vals[i];
+ hist_val = hist_field->fn(hist_field, rec);
+ atomic64_add(hist_val, &entry->sums[i]);
+ }
+
+ atomic64_inc(&entry->hitcount);
+}
+
+static void event_hist_trigger(struct event_trigger_data *data, void *rec)
+{
+ struct hist_trigger_data *hist_data = data->private_data;
+ unsigned long entries[HIST_STACKTRACE_DEPTH];
+ struct hist_trigger_entry *entry;
+ struct stack_trace stacktrace;
+ u64 field_contents;
+ void *key;
+
+ if (atomic64_read(&hist_data->drops)) {
+ atomic64_inc(&hist_data->drops);
+ return;
+ }
+
+ if (hist_data->key->flags & HIST_FIELD_STACKTRACE) {
+ stacktrace.max_entries = HIST_STACKTRACE_DEPTH;
+ stacktrace.entries = entries;
+ stacktrace.nr_entries = 0;
+ stacktrace.skip = HIST_STACKTRACE_SKIP;
+
+ memset(stacktrace.entries, 0, hist_data->map_attr.key_size);
+ save_stack_trace(&stacktrace);
+
+ key = entries;
+ } else {
+ field_contents = hist_data->key->fn(hist_data->key, rec);
+ if (hist_data->key->flags & HIST_FIELD_STRING)
+ key = (void *)field_contents;
+ else
+ key = (void *)&field_contents;
+ }
+
+ if (tracing_map_lookup_elem(hist_data->map, key, &entry) != 0) {
+ entry = hist_trigger_entry_create(hist_data);
+ if (!entry) {
+ atomic64_inc(&hist_data->drops);
+ return;
+ }
+ if (WARN_ON_ONCE(hist_trigger_entry_insert(hist_data,
+ key, entry))) {
+ hist_trigger_entry_free(entry);
+ return;
+ }
+ }
+ hist_trigger_entry_update(hist_data, entry, rec);
+
+ atomic64_inc(&hist_data->total_hits);
+}
+
+static void hist_trigger_stacktrace_print(struct seq_file *m,
+ unsigned long *stacktrace_entries,
+ unsigned int max_entries)
+{
+ char str[KSYM_SYMBOL_LEN];
+ unsigned int spaces = 8;
+ unsigned int i;
+
+ for (i = 0; i < max_entries; i++) {
+ if (stacktrace_entries[i] == ULONG_MAX)
+ return;
+
+ seq_printf(m, "%*c", 1 + spaces, ' ');
+ sprint_symbol(str, stacktrace_entries[i]);
+ seq_printf(m, "%s\n", str);
+ }
+}
+
+static void hist_trigger_entry_print(struct seq_file *m,
+ struct hist_trigger_data *hist_data,
+ void *key,
+ struct hist_trigger_entry *entry)
+{
+ char str[KSYM_SYMBOL_LEN];
+ unsigned int i;
+ u64 uval;
+
+ if (entry->hist_data->key->flags & HIST_FIELD_SYM) {
+ uval = *(u64 *)key;
+ kallsyms_lookup(uval, NULL, NULL, NULL, str);
+ seq_printf(m, "%s: [%llx] %-35s", hist_data->key->field->name,
+ uval, str);
+ } else if (entry->hist_data->key->flags & HIST_FIELD_HEX) {
+ uval = *(u64 *)key;
+ seq_printf(m, "%s: %llx", hist_data->key->field->name, uval);
+ } else if (entry->hist_data->key->flags & HIST_FIELD_STACKTRACE) {
+ seq_puts(m, "stacktrace:\n");
+ hist_trigger_stacktrace_print(m, key, HIST_STACKTRACE_DEPTH);
+ } else if (entry->hist_data->key->flags & HIST_FIELD_STRING) {
+ seq_printf(m, "%s: %-35s", hist_data->key->field->name,
+ (char *)key);
+ } else if (entry->hist_data->key->flags & HIST_FIELD_EXECNAME) {
+ uval = *(u64 *)key;
+ seq_printf(m, "%s: %-16s[%10llu]", hist_data->key->field->name,
+ entry->comm, uval);
+ } else if (entry->hist_data->key->flags & HIST_FIELD_SYSCALL) {
+ const char *syscall_name;
+
+ uval = *(u64 *)key;
+ syscall_name = get_syscall_name(uval);
+ if (!syscall_name)
+ syscall_name = "unknown_syscall";
+
+ seq_printf(m, "%s: %-30s", hist_data->key->field->name,
+ syscall_name);
+ } else {
+ uval = *(u64 *)key;
+ seq_printf(m, "%s: %10llu", hist_data->key->field->name, uval);
+ }
+
+ seq_printf(m, " hitcount: %10llu",
+ (u64)atomic64_read(&entry->hitcount));
+
+ for (i = 0; i < hist_data->n_vals; i++) {
+ seq_printf(m, " %s: %10llu", hist_data->vals[i]->field->name,
+ (u64)atomic64_read(&entry->sums[i]));
+ }
+
+ seq_puts(m, "\n");
+}
+
+static int print_entries(struct seq_file *m,
+ struct hist_trigger_data *hist_data)
+{
+ void *key, *next_key, *tmp_key = NULL;
+ struct hist_trigger_entry *entry;
+ int ret = 0;
+
+ key = kzalloc(hist_data->map_attr.key_size, GFP_KERNEL);
+ if (!key)
+ return -ENOMEM;
+
+ next_key = kzalloc(hist_data->map_attr.key_size, GFP_KERNEL);
+ if (!next_key) {
+ ret = -ENOMEM;
+ goto free;
+ }
+
+ while (tracing_map_get_next_key(hist_data->map, key, next_key) == 0) {
+ tracing_map_lookup_elem(hist_data->map, next_key, &entry);
+ hist_trigger_entry_print(m, hist_data, next_key, entry);
+ tmp_key = key;
+ key = next_key;
+ next_key = tmp_key;
+ }
+free:
+ kfree(key);
+ kfree(next_key);
+
+ return ret;
+}
+
+static int hist_show(struct seq_file *m, void *v)
+{
+ struct event_trigger_data *test, *data = NULL;
+ struct ftrace_event_file *event_file;
+ struct hist_trigger_data *hist_data;
+ int ret = 0;
+
+ mutex_lock(&event_mutex);
+
+ event_file = event_file_data(m->private);
+
+ if (unlikely(!event_file)) {
+ ret = -ENODEV;
+ goto out_unlock;
+ }
+
+ list_for_each_entry_rcu(test, &event_file->triggers, list) {
+ if (test->cmd_ops->trigger_type == ETT_EVENT_HIST) {
+ data = test;
+ break;
+ }
+ }
+
+ if (!data)
+ goto out_unlock;
+
+ seq_puts(m, "# trigger info: ");
+ data->ops->print(m, data->ops, data);
+ seq_puts(m, "\n");
+
+ hist_data = data->private_data;
+
+ ret = print_entries(m, hist_data);
+
+ seq_printf(m, "\nTotals:\n Hits: %lu\n Entries: %u\n Dropped: %lu\n",
+ atomic64_read(&hist_data->total_hits),
+ atomic_read(&hist_data->next_entry),
+ atomic64_read(&hist_data->drops));
+ out_unlock:
+ mutex_unlock(&event_mutex);
+
+ return ret;
+}
+
+static int event_hist_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, hist_show, file);
+}
+
+const struct file_operations event_hist_fops = {
+ .open = event_hist_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static const char *get_hist_field_flags(struct hist_field *hist_field)
+{
+ const char *flags_str = NULL;
+
+ if (hist_field->flags & HIST_FIELD_SYM)
+ flags_str = "sym";
+ else if (hist_field->flags & HIST_FIELD_HEX)
+ flags_str = "hex";
+ else if (hist_field->flags & HIST_FIELD_SYSCALL)
+ flags_str = "syscall";
+ else if (hist_field->flags & HIST_FIELD_EXECNAME)
+ flags_str = "execname";
+
+ return flags_str;
+}
+
+static void hist_field_print(struct seq_file *m, struct hist_field *hist_field)
+{
+ seq_printf(m, "%s", hist_field->field->name);
+ if (hist_field->flags) {
+ const char *flags_str = get_hist_field_flags(hist_field);
+
+ if (flags_str)
+ seq_printf(m, ".%s", flags_str);
+ }
+}
+
+static int event_hist_trigger_print(struct seq_file *m,
+ struct event_trigger_ops *ops,
+ struct event_trigger_data *data)
+{
+ struct hist_trigger_data *hist_data = data->private_data;
+ unsigned int i;
+
+ seq_puts(m, "hist:keys=");
+
+ if (hist_data->key->flags & HIST_FIELD_STACKTRACE)
+ seq_puts(m, "stacktrace");
+ else
+ hist_field_print(m, hist_data->key);
+
+ seq_puts(m, ":vals=");
+
+ for (i = 0; i < hist_data->n_vals; i++) {
+ if (i > 0)
+ seq_puts(m, ",");
+ hist_field_print(m, hist_data->vals[i]);
+ }
+
+ seq_printf(m, ":size=%u", (1 << hist_data->map_bits));
+
+ if (data->filter_str)
+ seq_printf(m, " if %s", data->filter_str);
+
+ if (data->paused)
+ seq_puts(m, " [paused]");
+ else
+ seq_puts(m, " [active]");
+
+ seq_putc(m, '\n');
+
+ return 0;
+}
+
+static void event_hist_trigger_free(struct event_trigger_ops *ops,
+ struct event_trigger_data *data)
+{
+ struct hist_trigger_data *hist_data = data->private_data;
+
+ if (WARN_ON_ONCE(data->ref <= 0))
+ return;
+
+ data->ref--;
+ if (!data->ref) {
+ destroy_hist_data(hist_data);
+ trigger_data_free(data);
+ }
+}
+
+static struct event_trigger_ops event_hist_trigger_ops = {
+ .func = event_hist_trigger,
+ .print = event_hist_trigger_print,
+ .init = event_trigger_init,
+ .free = event_hist_trigger_free,
+};
+
+static struct event_trigger_ops *event_hist_get_trigger_ops(char *cmd,
+ char *param)
+{
+ return &event_hist_trigger_ops;
+}
+
+static void hist_clear(struct event_trigger_data *data)
+{
+ struct hist_trigger_data *hist_data = data->private_data;
+ bool paused;
+
+ paused = data->paused;
+ data->paused = true;
+
+ tracing_map_clear(hist_data->map);
+
+ atomic_set(&hist_data->next_entry, 0);
+ atomic64_set(&hist_data->total_hits, 0);
+ atomic64_set(&hist_data->drops, 0);
+
+ data->paused = paused;
+}
+
+static int hist_register_trigger(char *glob, struct event_trigger_ops *ops,
+ struct event_trigger_data *data,
+ struct ftrace_event_file *file)
+{
+ struct hist_trigger_data *hist_data = data->private_data;
+ struct event_trigger_data *test;
+ int ret = 0;
+
+ list_for_each_entry_rcu(test, &file->triggers, list) {
+ if (test->cmd_ops->trigger_type == ETT_EVENT_HIST) {
+ if (hist_data->attrs->clear)
+ hist_clear(test);
+ else if (hist_data->attrs->pause)
+ test->paused = true;
+ else if (hist_data->attrs->cont)
+ test->paused = false;
+ else
+ ret = -EEXIST;
+ goto out;
+ }
+ }
+
+ if (hist_data->attrs->pause)
+ data->paused = true;
+
+ if (data->ops->init) {
+ ret = data->ops->init(data->ops, data);
+ if (ret < 0)
+ goto out;
+ }
+
+ list_add_rcu(&data->list, &file->triggers);
+ ret++;
+
+ if (trace_event_trigger_enable_disable(file, 1) < 0) {
+ list_del_rcu(&data->list);
+ ret--;
+ }
+ update_cond_flag(file);
+out:
+ return ret;
+}
+
+static int event_hist_trigger_func(struct event_command *cmd_ops,
+ struct ftrace_event_file *file,
+ char *glob, char *cmd, char *param)
+{
+ unsigned int hist_trigger_bits = HIST_TRIGGER_BITS;
+ struct event_trigger_data *trigger_data;
+ struct hist_trigger_attrs *attrs;
+ struct event_trigger_ops *trigger_ops;
+ struct hist_trigger_data *hist_data;
+ char *trigger;
+ int ret = 0;
+
+ if (!param)
+ return -EINVAL;
+
+ /* separate the trigger from the filter (k:v [if filter]) */
+ trigger = strsep(&param, " \t");
+ if (!trigger)
+ return -EINVAL;
+
+ attrs = parse_hist_trigger_attrs(trigger);
+ if (IS_ERR(attrs))
+ return PTR_ERR(attrs);
+
+ if (!attrs->keys_str || !attrs->vals_str)
+ return -EINVAL;
+
+ if (attrs->map_bits)
+ hist_trigger_bits = attrs->map_bits;
+
+ hist_data = create_hist_data(hist_trigger_bits, attrs, file);
+ if (IS_ERR(hist_data))
+ return PTR_ERR(hist_data);
+
+ trigger_ops = cmd_ops->get_trigger_ops(cmd, trigger);
+
+ ret = -ENOMEM;
+ trigger_data = kzalloc(sizeof(*trigger_data), GFP_KERNEL);
+ if (!trigger_data)
+ goto out;
+
+ trigger_data->count = -1;
+ trigger_data->ops = trigger_ops;
+ trigger_data->cmd_ops = cmd_ops;
+
+ INIT_LIST_HEAD(&trigger_data->list);
+ RCU_INIT_POINTER(trigger_data->filter, NULL);
+
+ trigger_data->private_data = hist_data;
+
+ if (glob[0] == '!') {
+ cmd_ops->unreg(glob+1, trigger_ops, trigger_data, file);
+ ret = 0;
+ goto out_free;
+ }
+
+ if (!param) /* if param is non-empty, it's supposed to be a filter */
+ goto out_reg;
+
+ if (!cmd_ops->set_filter)
+ goto out_reg;
+
+ ret = cmd_ops->set_filter(param, trigger_data, file);
+ if (ret < 0)
+ goto out_free;
+ out_reg:
+ ret = cmd_ops->reg(glob, trigger_ops, trigger_data, file);
+ /*
+ * The above returns on success the # of triggers registered,
+ * but if it didn't register any it returns zero. Consider no
+ * triggers registered a failure too.
+ */
+ if (!ret) {
+ if (!(attrs->pause || attrs->cont || attrs->clear))
+ ret = -ENOENT;
+ goto out_free;
+ } else if (ret < 0)
+ goto out_free;
+ /* Just return zero, not the number of registered triggers */
+ ret = 0;
+ out:
+ return ret;
+ out_free:
+ if (cmd_ops->set_filter)
+ cmd_ops->set_filter(NULL, trigger_data, NULL);
+ kfree(trigger_data);
+ destroy_hist_data(hist_data);
+
+ goto out;
+}
+
+static struct event_command trigger_hist_cmd = {
+ .name = "hist",
+ .trigger_type = ETT_EVENT_HIST,
+ .post_trigger = true, /* need non-NULL rec */
+ .func = event_hist_trigger_func,
+ .reg = hist_register_trigger,
+ .unreg = unregister_trigger,
+ .get_trigger_ops = event_hist_get_trigger_ops,
+ .set_filter = set_trigger_filter,
+};
+
+static __init int register_trigger_hist_cmd(void)
+{
+ int ret;
+
+ ret = register_event_command(&trigger_hist_cmd);
+ WARN_ON(ret < 0);
+
+ return ret;
+}
+
__init int register_trigger_cmds(void)
{
register_trigger_traceon_traceoff_cmds();
register_trigger_snapshot_cmd();
register_trigger_stacktrace_cmd();
register_trigger_enable_disable_cmds();
+ register_trigger_hist_cmd();

return 0;
}
--
1.9.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/