Re: [PATCH v4 4/4] perf test: Don't starve per-process tests with system-wide events

[Ian Rogers]

On Mon, Nov 9, 2026 at 5:48 AM James Clark <james.clark@xxxxxxxxxx> wrote:
>
> Some tests intermittently fail when run in parallel, for example "Check
> branch stack sampling" fails with no branches recorded at all.
>
> The default event is cycles:P which there may only be 1 or 2 HW counters
> for on Intel because it's precise. System wide or per-core precise
> events hog the HW counter leaving other per-process tests with no
> samples. Rather than make all the tests that use default record commands
> run serially, just make the system wide ones use a software event which
> has the greatest possible compatibility. They don't need precise
> sampling or HW counters anyway. The per-process tests can be left alone
> as their events/processes never run concurrently on any one core.
>
> The LBR filter is another resource shared by all events on a CPU, and
> the system-wide LBR test uses different filters to the per-process
> branch stack test, so mark the LBR test as exclusive.

Thanks for deflaking the tests! Making the LBR test exclusive is
pragmatic; however, it also feels like a shame since recording LBR
data in parallel should be something we expect users to do.

For PMUs that can have a single user we have logic which prints
conflicting users in dump_perf_event_processes:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/evsel.c?h=perf-tools-next#n4519

I wonder if, when we get zero counts or no samples we could perform a
similar analysis and report conflicting users—though this would
inherently be racy. If we had that, maybe we could just repeat the LBR
tests until there are no conflicting users and not need the exclusive
tag.

Anyway, doing all of that is a bigger project.

Reviewed-by: Ian Rogers <irogers@xxxxxxxxxx>

Thanks,
Ian

> This fixes the "Check branch stack sampling" test and other flakes when
> a system-wide record from another test runs at the same time, for
> example:
>
>   $ sudo perf record -a -o /dev/null -- sleep 30 &
>   $ sudo perf test -vvv "Check branch stack sampling"
>   ...
>   Testing user branch stack sampling
>   ERROR: Branches missing ^brstack_bench\+[^ ]*/brstack_foo\+[^ ]*/IND_CALL/.*$
>   ...
>   Testing user branch stack sampling [Failed]
>   ...
>   Testing branch stack filtering permutation (call,CALL|SYSCALL)
>   Testing branch stack filtering [Failed empty script output]
>   ...
>   ---- end(-1) ----
>   138: Check branch stack sampling : FAILED!
>
> Signed-off-by: James Clark <james.clark@xxxxxxxxxx>
> ---
>  tools/perf/tests/shell/record.sh          | 6 +++---
>  tools/perf/tests/shell/record_lbr.sh      | 2 +-
>  tools/perf/tests/shell/record_sideband.sh | 4 ++--
>  3 files changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
> index 01dc3e22fa73..e4789e2bcf43 100755
> --- a/tools/perf/tests/shell/record.sh
> +++ b/tools/perf/tests/shell/record.sh
> @@ -388,7 +388,7 @@ test_system_wide() {
>    echo "Basic --system-wide mode test"
>    local ret=0
>    perf_record_with_retry "${perfdata}" "check_system_wide" "perf test -w thloop" \
> -    -aB --synth=no || ret=$?
> +    -aB --synth=no -e cpu-clock || ret=$?
>    if [ $ret -eq 2 ]; then
>      echo "System-wide record [Skipped not supported]"
>      return
> @@ -471,7 +471,7 @@ test_cgroup() {
>    echo "Cgroup sampling test"
>    local ret=0
>    perf_record_with_retry "${perfdata}" "check_cgroup" "perf test -w thloop" \
> -    -aB --synth=cgroup --all-cgroups || ret=$?
> +    -aB --synth=cgroup --all-cgroups -e cpu-clock || ret=$?
>    if [ $ret -eq 2 ]; then
>      echo "Cgroup sampling [Skipped not supported]"
>      return
> @@ -491,7 +491,7 @@ test_uid() {
>    echo "Uid sampling test"
>    local ret=0
>    perf_record_with_retry "${perfdata}" "check_uid" "perf test -w thloop" \
> -    -aB --synth=no --uid "$(id -u)" || ret=$?
> +    -aB --synth=no --uid "$(id -u)" -e cpu-clock || ret=$?
>    if [ $ret -eq 2 ]; then
>      local logfile="${PERF_RECORD_LOGS[${#PERF_RECORD_LOGS[@]}-1]}"
>      if grep -q -E "libbpf.*EPERM|Access to performance monitoring" "$logfile" || \
> diff --git a/tools/perf/tests/shell/record_lbr.sh b/tools/perf/tests/shell/record_lbr.sh
> index 8d51afeb437b..6b6af59db871 100755
> --- a/tools/perf/tests/shell/record_lbr.sh
> +++ b/tools/perf/tests/shell/record_lbr.sh
> @@ -1,6 +1,6 @@
>  #!/bin/bash
>  # SPDX-License-Identifier: GPL-2.0
> -# perf record LBR tests
> +# perf record LBR tests (exclusive)
>
>  set -e
>
> diff --git a/tools/perf/tests/shell/record_sideband.sh b/tools/perf/tests/shell/record_sideband.sh
> index 2182551873be..f16889e3fa9a 100755
> --- a/tools/perf/tests/shell/record_sideband.sh
> +++ b/tools/perf/tests/shell/record_sideband.sh
> @@ -22,7 +22,7 @@ trap trap_cleanup EXIT TERM INT
>
>  can_cpu_wide()
>  {
> -    if ! perf record -o ${perfdata} -BN --no-bpf-event -C $1 true > /dev/null 2>&1
> +    if ! perf record -o ${perfdata} -BN --no-bpf-event -e cpu-clock -C $1 true > /dev/null 2>&1
>      then
>          echo "record sideband test [Skipped cannot record cpu$1]"
>          err=2
> @@ -39,7 +39,7 @@ test_system_wide_tracking()
>      can_cpu_wide 1 || return 0
>
>      # Record on CPU 0 a task running on CPU 1
> -    perf record -BN --no-bpf-event -o ${perfdata} -C 0 -- taskset --cpu-list 1 true
> +    perf record -BN --no-bpf-event -e cpu-clock -o ${perfdata} -C 0 -- taskset --cpu-list 1 true
>
>      # Should get MMAP events from CPU 1
>      mmap_cnt=`perf script -i ${perfdata} --show-mmap-events -C 1 2>/dev/null | grep MMAP | wc -l`
>
> --
> 2.43.0
>