[PATCH v8 0/4] perf sched latency: Refine outputs, unit scaling, and histogram support

[Aaron Tomlin]

Hi Namhyung, Ian, Arnaldo,

This patch series improves 'perf sched latency' by suppressing misleading
empty table output, extending pipe mode stream processing, introducing
dynamic unit auto-scaling for latency and runtime statistics, and adding
latency histogram visualisation alongside time-span filtering.

Patch 1 addresses an issue where 'perf sched latency' fell through and
returned success (0) when perf_session__has_traces() failed due to missing
tracepoint events in a perf.data file. This caused empty header tables and
zeroed summary statistics to be rendered. In addition,
thread__get_runtime() in map_switch_event() is guarded against potential
NULL pointer dereferences under memory allocation failures.

Patch 2 extends pipe mode stream support. Because event attributes in pipe
mode are received dynamically during event processing, session->evlist is
not populated prior to event processing.
To handle pipe input correctly:
    - Register the missing .attr, .tracing_data, .build_id, and .feature
      callbacks in cmd_sched()
    
    - Promote the handlers array to file-scope (latency_handlers[]) and
      dynamically assign matching tracepoint handlers (or
      process_sched_ignore) inside perf_sched__process_tracepoint_sample()
      when evsel->handler is NULL; replace process_sched_wakeup_ignore()
      with process_sched_ignore()
    
    - Perform the trace check post-processing when handling pipe data

Patch 3 introduces dynamic auto-scaling for latency and runtime display
columns (Runtime, Avg delay, Max delay). Previously, all values were
unconditionally formatted in milliseconds (ms), making microsecond- or
second-scale latencies difficult to read. Columns are now dynamically
scaled to the most appropriate unit (ns, us, ms, s), column headers are
updated, and format specifiers are aligned character-for-character with
table headers.

Patch 4 adds three new command-line options to 'perf sched latency':
    --histogram (-H):
            Displays an ASCII bar chart of CPU wait latencies between
            snapshots
    --hist-mode:
            Configures the bucketing scheme to either logarithmic (log) or
            100 us equal-width linear (linear) mode
    --time:
            Filters trace event processing to a specified [start,stop] time
            span

Changes since v7:

 - Added a guard clause in perf_sched__process_tracepoint_sample() to
   reduce loop indentation (Namhyung Kim)

 - Converted dummy process_sched_wakeup_ignore() references to
   process_sched_ignore and eliminated redundant code (Namhyung Kim)

 - Replaced string comparisons with faster integer checks (i.e.,
   thread__tid(...) != 0) for swapper (idle thread) handling (Namhyung Kim)

 - Added automated test cases for --histogram, --hist-mode, and --time into
   tools/perf/tests/shell/sched.sh (Namhyung Kim)

 - Link to v7: https://lore.kernel.org/lkml/20260802210914.199941-1-atomlin@xxxxxxxxxxx/

Changes since v6:

 - Fixed O(N) hot-path traversal overhead in
   perf_sched__process_tracepoint_sample() for unhandled tracepoints by
   assigning a dummy ignore handler (process_sched_ignore)

 - Replaced evlist__set_tracepoints_handlers() in sample processing with a
   per-evsel lookup to prevent -EEXIST early aborts and avoid dropping
   late-arriving pipe events

 - Updated commit message for the pipe mode patch to reflect the
   single-pass handler assignment logic

 - Link to v6: https://lore.kernel.org/lkml/20260801234008.176724-1-atomlin@xxxxxxxxxxx/

Changes since v5:

 - Split pipe mode trace sample handling from Patch 1 into a standalone
   patch (Namhyung Kim)

 - Added a Fixes: tag to the pipe mode patch referencing commit
   27295592c22e ("perf session: Share the common trace sample_check routine
   as perf_session__has_traces")

 - Updated commit log with before and after illustrations of table header
   formatting

 - Link to v5: https://lore.kernel.org/lkml/20260730185416.97166-1-atomlin@xxxxxxxxxxx/

Changes since v4:

 - Added the missing .feature callback to sched.tool in cmd_sched()

 - Fixed memory leak by reusing uncompleted work atoms when sched_in events
   are skipped or lost

 - Promoted the handlers array to file-scope (i.e., latency_handlers[]) and
   added a dynamic lookup via evlist__set_tracepoints_handlers() inside
   perf_sched__process_tracepoint_sample() whenever a sample arrives with
   an unattached handler (i.e., evsel->handler == NULL)

 - Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup)
   in pipe mode and non-pipe modes

 - Added missing trailing pipe in output_lat_thread's new auto-scaling
   format string

 - Stripped redundant prefix strings from each row's format string to
   produce a clean, tabular output

 - Link to v4: https://lore.kernel.org/lkml/20260729144451.38286-1-atomlin@xxxxxxxxxxx/

Changes since v3:

 - Registered Missing Callbacks. In perf_tool__init configuration inside
   cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks.
   Without these callbacks, pipe mode drops header attributes entirely,
   preventing tracepoints from being populated in session->evlist

 - Introduced an explicit post-processing pipe check. This ensures that
   pipe mode aborts correctly and does not produce superfluous empty
   latency tables when no trace samples are available

 - Prevented potential NULL pointer dereference. Added a NULL check for
   thread__get_runtime() in map_switch_event() under memory allocation
   failures

 - Fixed column alignment. Modified format specifiers to align
   character-for-character with header column widths across all rows

 - Adjusted header label padding. Updated header label padding and column
   width delimiters in perf_sched__lat() to match the exact field format
   specifiers printed in output_lat_thread()

 - Fixed swapper histogram inclusion. Excluded "swapper" (i.e.,
   CPU-specific idle thread) from global_hist buckets

 - Preserved task state machine across --time bounds. Moved time-window
   filtering inside add_sched_in_event() to gate latency statistics
   recording without breaking wakeup state tracking across time interval
   boundaries

 - Link to v3: https://lore.kernel.org/lkml/20260726032533.712462-1-atomlin@xxxxxxxxxxx/

Changes since v2:

 - Ensured vertical pipe separators align across all latency table columns

 - Excluded "swapper" (idle thread) latency samples from global_hist

 - Preserved task state machine transitions across --time boundaries

 - Suppressed empty table headers, total lines, and histogram graphs when
   no matching trace samples exist (e.g., when --time, --CPU, or --pids
   exclude all samples), outputting "No matching trace samples found."
   instead

 - Linked to v2: https://lore.kernel.org/lkml/20260725173341.679782-1-atomlin@xxxxxxxxxxx/

Changes since v1:

 - Fixed integer overflow in latency_bucket() (Ian Rogers)

 - Linked to v1: https://lore.kernel.org/lkml/20260724142901.634761-1-atomlin@xxxxxxxxxxx/

Aaron Tomlin (4):
  perf sched: Suppress latency table output when trace samples are
    missing
  perf sched: Handle missing trace samples in pipe mode
  perf sched latency: Auto-scale latency and runtime display units
  perf sched latency: Add histogram and time interval options

 tools/perf/Documentation/perf-sched.txt |   6 +
 tools/perf/builtin-sched.c              | 347 ++++++++++++++++++++----
 tools/perf/tests/shell/sched.sh         |  29 ++
 3 files changed, 332 insertions(+), 50 deletions(-)

-- 
2.55.0