Re: [PATCH v8 0/4] perf sched latency: Refine outputs, unit scaling, and histogram support

[Ian Rogers]

On Wed, Aug 5, 2026 at 2:08 PM Aaron Tomlin <atomlin@xxxxxxxxxxx> wrote:
>
> Hi Namhyung, Ian, Arnaldo,
>
> This patch series improves 'perf sched latency' by suppressing misleading
> empty table output, extending pipe mode stream processing, introducing
> dynamic unit auto-scaling for latency and runtime statistics, and adding
> latency histogram visualisation alongside time-span filtering.
>
> Patch 1 addresses an issue where 'perf sched latency' fell through and
> returned success (0) when perf_session__has_traces() failed due to missing
> tracepoint events in a perf.data file. This caused empty header tables and
> zeroed summary statistics to be rendered. In addition,
> thread__get_runtime() in map_switch_event() is guarded against potential
> NULL pointer dereferences under memory allocation failures.
>
> Patch 2 extends pipe mode stream support. Because event attributes in pipe
> mode are received dynamically during event processing, session->evlist is
> not populated prior to event processing.
> To handle pipe input correctly:
>     - Register the missing .attr, .tracing_data, .build_id, and .feature
>       callbacks in cmd_sched()
>
>     - Promote the handlers array to file-scope (latency_handlers[]) and
>       dynamically assign matching tracepoint handlers (or
>       process_sched_ignore) inside perf_sched__process_tracepoint_sample()
>       when evsel->handler is NULL; replace process_sched_wakeup_ignore()
>       with process_sched_ignore()
>
>     - Perform the trace check post-processing when handling pipe data
>
> Patch 3 introduces dynamic auto-scaling for latency and runtime display
> columns (Runtime, Avg delay, Max delay). Previously, all values were
> unconditionally formatted in milliseconds (ms), making microsecond- or
> second-scale latencies difficult to read. Columns are now dynamically
> scaled to the most appropriate unit (ns, us, ms, s), column headers are
> updated, and format specifiers are aligned character-for-character with
> table headers.
>
> Patch 4 adds three new command-line options to 'perf sched latency':
>     --histogram (-H):
>             Displays an ASCII bar chart of CPU wait latencies between
>             snapshots
>     --hist-mode:
>             Configures the bucketing scheme to either logarithmic (log) or
>             100 us equal-width linear (linear) mode
>     --time:
>             Filters trace event processing to a specified [start,stop] time
>             span

Thanks for these improvements and the tests for coverage! For the series:

Reviewed-by: Ian Rogers <irogers@xxxxxxxxxx>

Thanks,
Ian

> Changes since v7:
>
>  - Added a guard clause in perf_sched__process_tracepoint_sample() to
>    reduce loop indentation (Namhyung Kim)
>
>  - Converted dummy process_sched_wakeup_ignore() references to
>    process_sched_ignore and eliminated redundant code (Namhyung Kim)
>
>  - Replaced string comparisons with faster integer checks (i.e.,
>    thread__tid(...) != 0) for swapper (idle thread) handling (Namhyung Kim)
>
>  - Added automated test cases for --histogram, --hist-mode, and --time into
>    tools/perf/tests/shell/sched.sh (Namhyung Kim)
>
>  - Link to v7: https://lore.kernel.org/lkml/20260802210914.199941-1-atomlin@xxxxxxxxxxx/
>
> Changes since v6:
>
>  - Fixed O(N) hot-path traversal overhead in
>    perf_sched__process_tracepoint_sample() for unhandled tracepoints by
>    assigning a dummy ignore handler (process_sched_ignore)
>
>  - Replaced evlist__set_tracepoints_handlers() in sample processing with a
>    per-evsel lookup to prevent -EEXIST early aborts and avoid dropping
>    late-arriving pipe events
>
>  - Updated commit message for the pipe mode patch to reflect the
>    single-pass handler assignment logic
>
>  - Link to v6: https://lore.kernel.org/lkml/20260801234008.176724-1-atomlin@xxxxxxxxxxx/
>
> Changes since v5:
>
>  - Split pipe mode trace sample handling from Patch 1 into a standalone
>    patch (Namhyung Kim)
>
>  - Added a Fixes: tag to the pipe mode patch referencing commit
>    27295592c22e ("perf session: Share the common trace sample_check routine
>    as perf_session__has_traces")
>
>  - Updated commit log with before and after illustrations of table header
>    formatting
>
>  - Link to v5: https://lore.kernel.org/lkml/20260730185416.97166-1-atomlin@xxxxxxxxxxx/
>
> Changes since v4:
>
>  - Added the missing .feature callback to sched.tool in cmd_sched()
>
>  - Fixed memory leak by reusing uncompleted work atoms when sched_in events
>    are skipped or lost
>
>  - Promoted the handlers array to file-scope (i.e., latency_handlers[]) and
>    added a dynamic lookup via evlist__set_tracepoints_handlers() inside
>    perf_sched__process_tracepoint_sample() whenever a sample arrives with
>    an unattached handler (i.e., evsel->handler == NULL)
>
>  - Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup)
>    in pipe mode and non-pipe modes
>
>  - Added missing trailing pipe in output_lat_thread's new auto-scaling
>    format string
>
>  - Stripped redundant prefix strings from each row's format string to
>    produce a clean, tabular output
>
>  - Link to v4: https://lore.kernel.org/lkml/20260729144451.38286-1-atomlin@xxxxxxxxxxx/
>
> Changes since v3:
>
>  - Registered Missing Callbacks. In perf_tool__init configuration inside
>    cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks.
>    Without these callbacks, pipe mode drops header attributes entirely,
>    preventing tracepoints from being populated in session->evlist
>
>  - Introduced an explicit post-processing pipe check. This ensures that
>    pipe mode aborts correctly and does not produce superfluous empty
>    latency tables when no trace samples are available
>
>  - Prevented potential NULL pointer dereference. Added a NULL check for
>    thread__get_runtime() in map_switch_event() under memory allocation
>    failures
>
>  - Fixed column alignment. Modified format specifiers to align
>    character-for-character with header column widths across all rows
>
>  - Adjusted header label padding. Updated header label padding and column
>    width delimiters in perf_sched__lat() to match the exact field format
>    specifiers printed in output_lat_thread()
>
>  - Fixed swapper histogram inclusion. Excluded "swapper" (i.e.,
>    CPU-specific idle thread) from global_hist buckets
>
>  - Preserved task state machine across --time bounds. Moved time-window
>    filtering inside add_sched_in_event() to gate latency statistics
>    recording without breaking wakeup state tracking across time interval
>    boundaries
>
>  - Link to v3: https://lore.kernel.org/lkml/20260726032533.712462-1-atomlin@xxxxxxxxxxx/
>
> Changes since v2:
>
>  - Ensured vertical pipe separators align across all latency table columns
>
>  - Excluded "swapper" (idle thread) latency samples from global_hist
>
>  - Preserved task state machine transitions across --time boundaries
>
>  - Suppressed empty table headers, total lines, and histogram graphs when
>    no matching trace samples exist (e.g., when --time, --CPU, or --pids
>    exclude all samples), outputting "No matching trace samples found."
>    instead
>
>  - Linked to v2: https://lore.kernel.org/lkml/20260725173341.679782-1-atomlin@xxxxxxxxxxx/
>
> Changes since v1:
>
>  - Fixed integer overflow in latency_bucket() (Ian Rogers)
>
>  - Linked to v1: https://lore.kernel.org/lkml/20260724142901.634761-1-atomlin@xxxxxxxxxxx/
>
> Aaron Tomlin (4):
>   perf sched: Suppress latency table output when trace samples are
>     missing
>   perf sched: Handle missing trace samples in pipe mode
>   perf sched latency: Auto-scale latency and runtime display units
>   perf sched latency: Add histogram and time interval options
>
>  tools/perf/Documentation/perf-sched.txt |   6 +
>  tools/perf/builtin-sched.c              | 347 ++++++++++++++++++++----
>  tools/perf/tests/shell/sched.sh         |  29 ++
>  3 files changed, 332 insertions(+), 50 deletions(-)
>
> --
> 2.55.0
>