Re: [PATCH v3 5/5] perf record off-cpu: Add direct off-cpu test

[Howard Chu]

Hello Ian,

Thanks for reviewing this patch.

On Sat, Jul 27, 2024 at 8:54 AM Ian Rogers <irogers@xxxxxxxxxx> wrote:
>
> On Fri, Jul 26, 2024 at 3:28 AM Howard Chu <howardchu95@xxxxxxxxx> wrote:
> >
> > Add a simple workload(offcpu.c) to create the scenario for direct
> > off-cpu dumping.
> >
> > Signed-off-by: Howard Chu <howardchu95@xxxxxxxxx>
> > Suggested-by: Ian Rogers <irogers@xxxxxxxxxx>
>
> I tried with and without --off-cpu-thresh=1 but I only see 1
> offcpu-time sample and no offcpu-time-direct.

TL;DR:

Please use 'perf record -F 1 -D 999 --off-cpu --off-cpu-thresh 1999
perf test -w offcpu' :)

We need to give it an initial delay, using '-D 999'.

That's because when using "perf record <workload>", evlist is not
enabled if you don't give it an initial delay or specify cpu. There
are two occurrences of evlist__enable() in builtin-record.c:

if (!target__none(&opts->target) && !opts->target.initial_delay)
    evlist__enable(rec->evlist);

Here, if you perf record a workload, say "perf record --off-cpu
--off-cpu-thresh=1 perf test -w offcpu", the
"!target__none(&opts->target)" will always be 0, so no enabling.

if (opts->target.initial_delay) {
    pr_info(EVLIST_DISABLED_MSG);
    if (opts->target.initial_delay > 0) {
        usleep(opts->target.initial_delay * USEC_PER_MSEC);
        evlist__enable(rec->evlist);
        pr_info(EVLIST_ENABLED_MSG);
    }
}

But we can do '-D 999' to give it an initial delay, so the evlist does
get enabled, now BPF can perf_event_output() a direct off-cpu sample
to the evlist. This is also the approach I used in the
'tests/shell/record_offcpu.sh'.
```
  if ! perf record -F 1 -D 999 --off-cpu --off-cpu-thresh 1999 -o
${perfdata} ${TEST_PROGRAM} 2> /dev/null
```

Thanks,
Howard
>
> Thanks,
> Ian
>
> > ---
> >  tools/perf/tests/builtin-test.c         |  1 +
> >  tools/perf/tests/shell/record_offcpu.sh | 29 +++++++++++++++++++++++++
> >  tools/perf/tests/tests.h                |  1 +
> >  tools/perf/tests/workloads/Build        |  1 +
> >  tools/perf/tests/workloads/offcpu.c     | 16 ++++++++++++++
> >  5 files changed, 48 insertions(+)
> >  create mode 100644 tools/perf/tests/workloads/offcpu.c
> >
> > diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
> > index c3d84b67ca8e..5062058ad17d 100644
> > --- a/tools/perf/tests/builtin-test.c
> > +++ b/tools/perf/tests/builtin-test.c
> > @@ -152,6 +152,7 @@ static struct test_workload *workloads[] = {
> >         &workload__sqrtloop,
> >         &workload__brstack,
> >         &workload__datasym,
> > +       &workload__offcpu,
> >  };
> >
> >  static int num_subtests(const struct test_suite *t)
> > diff --git a/tools/perf/tests/shell/record_offcpu.sh b/tools/perf/tests/shell/record_offcpu.sh
> > index 67c925f3a15a..1ea0a44336e2 100755
> > --- a/tools/perf/tests/shell/record_offcpu.sh
> > +++ b/tools/perf/tests/shell/record_offcpu.sh
> > @@ -6,6 +6,7 @@ set -e
> >
> >  err=0
> >  perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
> > +TEST_PROGRAM="perf test -w offcpu"
> >
> >  cleanup() {
> >    rm -f ${perfdata}
> > @@ -88,6 +89,30 @@ test_offcpu_child() {
> >    echo "Child task off-cpu test [Success]"
> >  }
> >
> > +test_offcpu_direct() {
> > +  echo "Direct off-cpu test"
> > +  # dump off-cpu samples for tasks blocked for more than 1999ms (1.9s)
> > +  # -D for initial delay, which is necessary if we want to enable evlist
> > +  if ! perf record -F 1 -D 999 --off-cpu --off-cpu-thresh 1999 -o ${perfdata} ${TEST_PROGRAM} 2> /dev/null
> > +  then
> > +    echo "Direct off-cpu test [Failed record]"
> > +    err=1
> > +    return
> > +  fi
> > +  if ! perf evlist -i ${perfdata} | grep -q "offcpu-time-direct"
> > +  then
> > +    echo "Direct off-cpu test [Failed no event]"
> > +    err=1
> > +    return
> > +  fi
> > +  if ! perf script -i ${perfdata} | grep -q -E ".*2[0-9]{9}[ ]*offcpu-time-direct" # 2 seconds (2,000,000,000)
> > +  then
> > +    echo "Direct off-cpu test [Failed missing output]"
> > +    err=1
> > +    return
> > +  fi
> > +  echo "Direct off-cpu test [Success]"
> > +}
> >
> >  test_offcpu_priv
> >
> > @@ -99,5 +124,9 @@ if [ $err = 0 ]; then
> >    test_offcpu_child
> >  fi
> >
> > +if [ $err = 0 ]; then
> > +  test_offcpu_direct
> > +fi
> > +
> >  cleanup
> >  exit $err
> > diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
> > index 3aa7701ee0e9..84ab15683269 100644
> > --- a/tools/perf/tests/tests.h
> > +++ b/tools/perf/tests/tests.h
> > @@ -205,6 +205,7 @@ DECLARE_WORKLOAD(leafloop);
> >  DECLARE_WORKLOAD(sqrtloop);
> >  DECLARE_WORKLOAD(brstack);
> >  DECLARE_WORKLOAD(datasym);
> > +DECLARE_WORKLOAD(offcpu);
> >
> >  extern const char *dso_to_test;
> >  extern const char *test_objdump_path;
> > diff --git a/tools/perf/tests/workloads/Build b/tools/perf/tests/workloads/Build
> > index 48bf0d3b0f3d..f37e9be8b142 100644
> > --- a/tools/perf/tests/workloads/Build
> > +++ b/tools/perf/tests/workloads/Build
> > @@ -6,6 +6,7 @@ perf-test-y += leafloop.o
> >  perf-test-y += sqrtloop.o
> >  perf-test-y += brstack.o
> >  perf-test-y += datasym.o
> > +perf-test-y += offcpu.o
> >
> >  CFLAGS_sqrtloop.o         = -g -O0 -fno-inline -U_FORTIFY_SOURCE
> >  CFLAGS_leafloop.o         = -g -O0 -fno-inline -fno-omit-frame-pointer -U_FORTIFY_SOURCE
> > diff --git a/tools/perf/tests/workloads/offcpu.c b/tools/perf/tests/workloads/offcpu.c
> > new file mode 100644
> > index 000000000000..02be3d05b06d
> > --- /dev/null
> > +++ b/tools/perf/tests/workloads/offcpu.c
> > @@ -0,0 +1,16 @@
> > +#include <linux/compiler.h>
> > +#include <unistd.h>
> > +#include "../tests.h"
> > +
> > +static int offcpu(int argc __maybe_unused, const char **argv __maybe_unused)
> > +{
> > +       /* get pass initial delay */
> > +       sleep(1);
> > +
> > +       /* what we want to collect as a direct sample */
> > +       sleep(2);
> > +
> > +       return 0;
> > +}
> > +
> > +DEFINE_WORKLOAD(offcpu);
> > --
> > 2.45.2
> >