[PATCH V5 00/16] perf tools: Introduce an abstraction for AUX Area and Instruction Tracing

[Adrian Hunter]

Hi

Here is V5 patches for the introduction of an abstraction for
using the AUX area and Instruction tracing. The patches for
AUX area support have been applied, just leaving patches for
Intel PT and Intel BTS.

A new patch "perf tools: Disallow PMU events intel_pt and
intel_bts until there is support" could go to v4.1 because
that is when intel_pt and intel_bts PMU's can show up.

The patches can also be found here:

	http://git.infradead.org/users/ahunter/linux-perf.git

An example (unchanged from V3) perf.data file and build id archive
can be found here:

	http://git.infradead.org/~ahunter/tfr/

There is also a tar of the 3 most relevant files with debugging
symbols. These need to be placed in under the correct paths in
/usr/lib/debug to get symbols.

Changes in V5:

   Patches already applied:
      perf report: Fix placement of itrace option in documentation
      perf tools: Add AUX area tracing index
      perf tools: Hit all build ids when AUX area tracing
      perf tools: Add build option NO_AUXTRACE to exclude AUX area tracing
      perf auxtrace: Add option to synthesize events for transactions
      perf tools: Add support for PERF_RECORD_AUX
      perf tools: Add support for PERF_RECORD_ITRACE_START
      perf tools: Add AUX area tracing Snapshot Mode
      perf record: Add AUX area tracing Snapshot Mode support

   perf tools: Disallow PMU events intel_pt and intel_bts until there is support
      New patch

   perf tools: Add Intel PT decoder
      Style improvements pointed out by Acme: aligning '=', single line initializing
      Make use of zalloc() not malloc / memset
      Make use of zfree
      Map internal error codes to fixed constants for output
      Change intel_pt_error_message() to intel_pt__strerror()

   perf tools: Add Intel PT support
      Make use of zfree

   perf tools: Take Intel PT into use
      Allow "intel_pt" PMU to be selected as an event

   perf tools: Add Intel BTS support
      Allow "intel_bts" PMU to be selected as an event
      Make use of zfree
      Map internal error codes to fixed constants for output
      Let "intel_bts" show up in 'perf list'

   perf tools: Output sample flags and insn_len from intel_bts
      Map internal error codes to fixed constants for output

Changes on V4:

   perf tools: Amend mmap ref counting for the AUX area mmap
      Dropped because already applied

   perf script: Always allow fields 'addr' and 'cpu' for auxtrace
      Dropped because already applied

   perf report: Add Instruction Tracing support
      Dropped because already applied

   perf report: Fix placement of itrace option in documentation
      New patch

   perf tools: Add AUX area tracing index
      Change size checks for more flexibility i.e.
      - don't mind if an indexed auxtrace_event is bigger than
      struct auxtrace_event
      - don't mind if the auxtrace index does not fill the whole
      file section
      Rename 'index' variable to 'ent' to avoid build errors on
      older gcc

   perf tools: Add build option NO_AUXTRACE to exclude AUX area tracing
      Fix whitespace alignment of NO_AUXTRACE=1
      Add NO_AUXTRACE=1 to make_minimal

   perf tools: Add support for PERF_RECORD_AUX
      Expand commit message

   perf tools: Add AUX area tracing Snapshot Mode
      Whitespace fixups

   perf record: Add AUX area tracing Snapshot Mode support
      Whitespace fixups
      Don't init static variables to 0 or NULL

   perf tools: Add Intel PT packet decoder
      Whitespace fixups

   perf tools: Add Intel PT instruction decoder
      Avoid build error on older (broken) gcc by adding -Wno-override-init
      Avoid build errors due to funny collate sequences i.e. use LC_COLLATE=C etc

   perf tools: Add Intel PT decoder
      Avoid build errors initializing structures to 0

   perf tools: Add Intel PT support
      Avoid build errors initializing structures to 0
      Allow for perf_pmu__config_terms() having an extra parameter now
      Allow for parse_events() having an extra parameter now
      Rename 'div' variable to 'd' to avoid build errors
      Whitespace fixup
      Remove a couple of unused enums

   perf tools: Add Intel BTS support
      Avoid build errors initializing structures to 0
      Allow for parse_events() having an extra parameter now

   perf tools: Put itrace options into an asciidoc include
      New patch

Changes in V3:

   New patch:
      perf tools: Amend mmap ref counting for the AUX area mmap

   Move some code under arch:
      perf tools: Add Intel PT support
      perf tools: Add Intel BTS support

   Updated documentation:
      perf report: Add Instruction Tracing support
      perf auxtrace: Add option to synthesize events for transactions
      perf tools: Take Intel PT into use
      perf tools: Add Intel BTS support

   Patches already applied:
      perf header: Add AUX area tracing feature
      perf evlist: Add support for mmapping an AUX area buffer
      perf tools: Add user events for AUX area tracing
      perf tools: Add support for AUX area recording
      perf record: Add basic AUX area tracing support
      perf record: Extend -m option for AUX area tracing mmap pages
      perf tools: Add a user event for AUX area tracing errors
      perf session: Add hooks to allow transparent decoding of AUX area tracing data
      perf session: Add instruction tracing options
      perf auxtrace: Add helpers for AUX area tracing errors
      perf auxtrace: Add helpers for queuing AUX area tracing data
      perf auxtrace: Add a heap for sorting AUX area tracing queues
      perf auxtrace: Add processing for AUX area tracing events
      perf auxtrace: Add a hashtable for caching
      perf tools: Add member to struct dso for an instruction cache
      perf script: Add Instruction Tracing support
      perf inject: Re-pipe AUX area tracing events
      perf inject: Add Instruction Tracing support
      perf script: Add field option 'flags' to print sample flags
      perf tools: Add aux_watermark member of struct perf_event_attr

Changes in V2:

   Get rid of MIN()
      perf auxtrace: Add helpers for AUX area tracing errors
      perf inject: Re-pipe AUX area tracing events
      perf tools: Add build option NO_AUXTRACE to exclude AUX area tracing


Intel BTS can be used on most recent Intel CPUs. Intel PT
is available on Broadwell.

Examples:

	Trace 'ls' with Intel BTS userspace only

	perf record --per-thread -e intel_bts//u ls
	perf report
	perf script

	Trace 'ls' with Intel BTS kernel and userspace

	~/libexec/perf-core/perf-with-kcore record bts-ls --per-thread -e intel_bts// -- ls
	~/libexec/perf-core/perf-with-kcore report bts-ls
	~/libexec/perf-core/perf-with-kcore script bts-ls

	Trace 'ls' with Intel PT userspace only

	perf record -e intel_pt//u ls
	perf report
	perf script

	Trace 'ls' with Intel PT kernel and userspace

	~/libexec/perf-core/perf-with-kcore record pt-ls -e intel_pt// -- ls
	~/libexec/perf-core/perf-with-kcore report pt-ls
	~/libexec/perf-core/perf-with-kcore script pt-ls


The abstraction has two separate aspects:
	1. recording AUX area data
	2. processing AUX area data

Recording consists of mmapping a separate buffer and copying
the data into the perf.data file.  The buffer is an AUX area
buffer.  The data is written preceded by a new user event
PERF_RECORD_AUXTRACE.  The data is too big to fit in the event
but follows immediately afterward. Session processing has to
skip to get to the next event header in a similar fashion to
the existing PERF_RECORD_HEADER_TRACING_DATA
event.  The main recording patches are:

      perf evlist: Add support for mmapping an AUX area buffer
      perf tools: Add user events for AUX area tracing
      perf tools: Add support for AUX area recording
      perf record: Add basic AUX area tracing support

Processing consists of providing hooks in session processing
to enable a decoder to see all the events and deliver synthesized
events transparently into the event stream.  The main processing
patch is:

      perf session: Add hooks to allow transparent decoding of AUX area tracing data


Adrian Hunter (16):
      perf tools: Disallow PMU events intel_pt and intel_bts until there is support
      perf auxtrace: Add Intel PT as an AUX area tracing type
      perf tools: Add Intel PT packet decoder
      perf tools: Add Intel PT instruction decoder
      perf tools: Add Intel PT log
      perf tools: Add Intel PT decoder
      perf tools: Add Intel PT support
      perf tools: Take Intel PT into use
      perf tools: Allow auxtrace data alignment
      perf tools: Add Intel BTS support
      perf tools: Output sample flags and insn_len from intel_pt
      perf tools: Output sample flags and insn_len from intel_bts
      perf tools: Intel PT to always update thread stack trace number
      perf tools: Intel BTS to always update thread stack trace number
      perf tools: Add example call-graph script
      perf tools: Put itrace options into an asciidoc include

 tools/build/Makefile.build                         |    2 +
 tools/perf/.gitignore                              |    2 +
 tools/perf/Documentation/intel-bts.txt             |   69 +
 tools/perf/Documentation/intel-pt.txt              |  467 +++++
 tools/perf/Documentation/itrace.txt                |   22 +
 tools/perf/Documentation/perf-inject.txt           |   23 +-
 tools/perf/Documentation/perf-report.txt           |   23 +-
 tools/perf/Documentation/perf-script.txt           |   23 +-
 tools/perf/Makefile.perf                           |   12 +-
 tools/perf/arch/x86/util/Build                     |    5 +
 tools/perf/arch/x86/util/auxtrace.c                |   83 +
 tools/perf/arch/x86/util/intel-bts.c               |  458 +++++
 tools/perf/arch/x86/util/intel-pt.c                |  752 ++++++++
 tools/perf/arch/x86/util/pmu.c                     |   18 +
 .../scripts/python/call-graph-from-postgresql.py   |  285 +++
 tools/perf/util/Build                              |    3 +
 tools/perf/util/auxtrace.c                         |   16 +-
 tools/perf/util/auxtrace.h                         |    3 +
 tools/perf/util/intel-bts.c                        |  921 ++++++++++
 tools/perf/util/intel-bts.h                        |   43 +
 tools/perf/util/intel-pt-decoder/Build             |   14 +
 .../perf/util/intel-pt-decoder/intel-pt-decoder.c  | 1758 ++++++++++++++++++
 .../perf/util/intel-pt-decoder/intel-pt-decoder.h  |  102 ++
 .../util/intel-pt-decoder/intel-pt-insn-decoder.c  |  246 +++
 .../util/intel-pt-decoder/intel-pt-insn-decoder.h  |   65 +
 tools/perf/util/intel-pt-decoder/intel-pt-log.c    |  155 ++
 tools/perf/util/intel-pt-decoder/intel-pt-log.h    |   52 +
 .../util/intel-pt-decoder/intel-pt-pkt-decoder.c   |  400 +++++
 .../util/intel-pt-decoder/intel-pt-pkt-decoder.h   |   64 +
 tools/perf/util/intel-pt.c                         | 1895 ++++++++++++++++++++
 tools/perf/util/intel-pt.h                         |   51 +
 31 files changed, 7963 insertions(+), 69 deletions(-)
 create mode 100644 tools/perf/Documentation/intel-bts.txt
 create mode 100644 tools/perf/Documentation/intel-pt.txt
 create mode 100644 tools/perf/Documentation/itrace.txt
 create mode 100644 tools/perf/arch/x86/util/auxtrace.c
 create mode 100644 tools/perf/arch/x86/util/intel-bts.c
 create mode 100644 tools/perf/arch/x86/util/intel-pt.c
 create mode 100644 tools/perf/arch/x86/util/pmu.c
 create mode 100644 tools/perf/scripts/python/call-graph-from-postgresql.py
 create mode 100644 tools/perf/util/intel-bts.c
 create mode 100644 tools/perf/util/intel-bts.h
 create mode 100644 tools/perf/util/intel-pt-decoder/Build
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-decoder.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-log.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-log.h
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.c
 create mode 100644 tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.h
 create mode 100644 tools/perf/util/intel-pt.c
 create mode 100644 tools/perf/util/intel-pt.h


Regards
Adrian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/