Re: [PATCH 4/4] perf test: Don't starve per-process tests with system-wide events

[Namhyung Kim]

On Thu, Nov 05, 2026 at 03:49:06PM +0000, James Clark wrote:
> Some tests intermittently fail when run in parallel, for example "Check
> branch stack sampling" fails with no branches recorded at all.
> 
> The default event is cycles:P which there may only be 1 or 2 HW counters
> for on Intel because it's precise. System wide or per-core precise
> events hog the HW counter leaving other per-process tests with no
> samples. Rather than make all the tests that use default record commands
> run serially, just make the system wide ones use a non-precise event.
> They don't need precise sampling anyway, so use plain cycles which can
> go on any counter. The per-process tests can be left alone as their
> events/processes never run concurrently on any one core.

One side effect of using default event is that it can fallback when it's
not available like on a VM or non-root.  Maybe better to use a software
event like cpu-clock instead.

Thanks,
Namhyung

> 
> The LBR filter is another resource shared by all events on a CPU, and
> the system-wide LBR test uses different filters to the per-process
> branch stack test, so mark the LBR test as exclusive.
> 
> This fixes the "Check branch stack sampling" test and other flakes when
> a system-wide record from another test runs at the same time, for
> example:
> 
>   $ sudo perf record -a -o /dev/null -- sleep 30 &
>   $ sudo perf test -vvv "Check branch stack sampling"
>   ...
>   Testing user branch stack sampling
>   ERROR: Branches missing ^brstack_bench\+[^ ]*/brstack_foo\+[^ ]*/IND_CALL/.*$
>   ...
>   Testing user branch stack sampling [Failed]
>   ...
>   Testing branch stack filtering permutation (call,CALL|SYSCALL)
>   Testing branch stack filtering [Failed empty script output]
>   ...
>   ---- end(-1) ----
>   138: Check branch stack sampling : FAILED!
> 
> Signed-off-by: James Clark <james.clark@xxxxxxxxxx>
> ---
>  tools/perf/tests/shell/record.sh          | 6 +++---
>  tools/perf/tests/shell/record_lbr.sh      | 2 +-
>  tools/perf/tests/shell/record_sideband.sh | 4 ++--
>  3 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
> index 01dc3e22fa73..5292eff78a70 100755
> --- a/tools/perf/tests/shell/record.sh
> +++ b/tools/perf/tests/shell/record.sh
> @@ -388,7 +388,7 @@ test_system_wide() {
>    echo "Basic --system-wide mode test"
>    local ret=0
>    perf_record_with_retry "${perfdata}" "check_system_wide" "perf test -w thloop" \
> -    -aB --synth=no || ret=$?
> +    -aB --synth=no -e cycles || ret=$?
>    if [ $ret -eq 2 ]; then
>      echo "System-wide record [Skipped not supported]"
>      return
> @@ -471,7 +471,7 @@ test_cgroup() {
>    echo "Cgroup sampling test"
>    local ret=0
>    perf_record_with_retry "${perfdata}" "check_cgroup" "perf test -w thloop" \
> -    -aB --synth=cgroup --all-cgroups || ret=$?
> +    -aB --synth=cgroup --all-cgroups -e cycles || ret=$?
>    if [ $ret -eq 2 ]; then
>      echo "Cgroup sampling [Skipped not supported]"
>      return
> @@ -491,7 +491,7 @@ test_uid() {
>    echo "Uid sampling test"
>    local ret=0
>    perf_record_with_retry "${perfdata}" "check_uid" "perf test -w thloop" \
> -    -aB --synth=no --uid "$(id -u)" || ret=$?
> +    -aB --synth=no --uid "$(id -u)" -e cycles || ret=$?
>    if [ $ret -eq 2 ]; then
>      local logfile="${PERF_RECORD_LOGS[${#PERF_RECORD_LOGS[@]}-1]}"
>      if grep -q -E "libbpf.*EPERM|Access to performance monitoring" "$logfile" || \
> diff --git a/tools/perf/tests/shell/record_lbr.sh b/tools/perf/tests/shell/record_lbr.sh
> index 8d51afeb437b..6b6af59db871 100755
> --- a/tools/perf/tests/shell/record_lbr.sh
> +++ b/tools/perf/tests/shell/record_lbr.sh
> @@ -1,6 +1,6 @@
>  #!/bin/bash
>  # SPDX-License-Identifier: GPL-2.0
> -# perf record LBR tests
> +# perf record LBR tests (exclusive)
>  
>  set -e
>  
> diff --git a/tools/perf/tests/shell/record_sideband.sh b/tools/perf/tests/shell/record_sideband.sh
> index 2182551873be..31ec2f5cde69 100755
> --- a/tools/perf/tests/shell/record_sideband.sh
> +++ b/tools/perf/tests/shell/record_sideband.sh
> @@ -22,7 +22,7 @@ trap trap_cleanup EXIT TERM INT
>  
>  can_cpu_wide()
>  {
> -    if ! perf record -o ${perfdata} -BN --no-bpf-event -C $1 true > /dev/null 2>&1
> +    if ! perf record -o ${perfdata} -BN --no-bpf-event -e cycles -C $1 true > /dev/null 2>&1
>      then
>          echo "record sideband test [Skipped cannot record cpu$1]"
>          err=2
> @@ -39,7 +39,7 @@ test_system_wide_tracking()
>      can_cpu_wide 1 || return 0
>  
>      # Record on CPU 0 a task running on CPU 1
> -    perf record -BN --no-bpf-event -o ${perfdata} -C 0 -- taskset --cpu-list 1 true
> +    perf record -BN --no-bpf-event -e cycles -o ${perfdata} -C 0 -- taskset --cpu-list 1 true
>  
>      # Should get MMAP events from CPU 1
>      mmap_cnt=`perf script -i ${perfdata} --show-mmap-events -C 1 2>/dev/null | grep MMAP | wc -l`
> 
> -- 
> 2.43.0
>