Re: [PATCH v3] perf trace: Implement syscall summary in BPF
From: Howard Chu
Date: Mon Mar 24 2025 - 00:48:09 EST
Hello,
On Sun, Mar 23, 2025 at 6:49 PM Howard Chu <howardchu95@xxxxxxxxx> wrote:
>
> Hello Namhyung,
>
> On Fri, Mar 21, 2025 at 11:42:55AM -0700, Namhyung Kim wrote:
> > When -s/--summary option is used, it doesn't need (augmented) arguments
> > of syscalls. Let's skip the augmentation and load another small BPF
> > program to collect the statistics in the kernel instead of copying the
> > data to the ring-buffer to calculate the stats in userspace. This will
> > be much more light-weight than the existing approach and remove any lost
> > events.
> >
> > Let's add a new option --bpf-summary to control this behavior. I cannot
> > make it default because there's no way to get e_machine in the BPF which
> > is needed for detecting different ABIs like 32-bit compat mode.
> >
> > No functional changes intended except for no more LOST events. :) But
> > it only works with -a/--all-cpus for now.
> >
> > $ sudo ./perf trace -as --summary-mode=total --bpf-summary sleep 1
> >
> > Summary of events:
> >
> > total, 6194 events
> >
> > syscall calls errors total min avg max stddev
> > (msec) (msec) (msec) (msec) (%)
> > --------------- -------- ------ -------- --------- --------- --------- ------
> > epoll_wait 561 0 4530.843 0.000 8.076 520.941 18.75%
> > futex 693 45 4317.231 0.000 6.230 500.077 21.98%
> > poll 300 0 1040.109 0.000 3.467 120.928 17.02%
> > clock_nanosleep 1 0 1000.172 1000.172 1000.172 1000.172 0.00%
> > ppoll 360 0 872.386 0.001 2.423 253.275 41.91%
> > epoll_pwait 14 0 384.349 0.001 27.453 380.002 98.79%
> > pselect6 14 0 108.130 7.198 7.724 8.206 0.85%
> > nanosleep 39 0 43.378 0.069 1.112 10.084 44.23%
> > ...
> >
> > Cc: Howard Chu <howardchu95@xxxxxxxxx>
> > Signed-off-by: Namhyung Kim <namhyung@xxxxxxxxxx>
> > ---
> > v3)
> > * support -S/--with-summary option too (Howard)
>
> It gave me segfault somehow.
>
> (gdb) bt
> #0 sighandler_dump_stack (sig=32767) at util/debug.c:322
> #1 <signal handler called>
> #2 0x00005555556d2383 in hashmap_find ()
> #3 0x000055555567474a in thread__update_stats (thread=0x5555564acc60, ttrace=0x5555564ad7a0, id=0, sample=0x7fffffff8f10, err=1,
> trace=0x7fffffffb1a0) at builtin-trace.c:2616
> #4 0x00005555556757cb in trace__sys_exit (trace=0x7fffffffb1a0, evsel=0x5555561879d0, event=0x7fffed980000, sample=0x7fffffff8f10)
> at builtin-trace.c:2924
> #5 0x0000555555677e1b in trace__handle_event (trace=0x7fffffffb1a0, event=0x7fffed980000, sample=0x7fffffff8f10) at builtin-trace.c:3619
> #6 0x00005555556796fb in __trace__deliver_event (trace=0x7fffffffb1a0, event=0x7fffed980000) at builtin-trace.c:4173
> #7 0x0000555555679859 in trace__deliver_event (trace=0x7fffffffb1a0, event=0x7fffed980000) at builtin-trace.c:4201
> #8 0x000055555567abed in trace__run (trace=0x7fffffffb1a0, argc=2, argv=0x7fffffffeb30) at builtin-trace.c:4590
> #9 0x000055555567f102 in cmd_trace (argc=2, argv=0x7fffffffeb30) at builtin-trace.c:5803
> #10 0x0000555555685252 in run_builtin (p=0x5555560eaf28 <commands+648>, argc=7, argv=0x7fffffffeb30) at perf.c:351
> #11 0x00005555556854fd in handle_internal_command (argc=7, argv=0x7fffffffeb30) at perf.c:404
> #12 0x000055555568565e in run_argv (argcp=0x7fffffffe91c, argv=0x7fffffffe910) at perf.c:448
> #13 0x00005555556859af in main (argc=7, argv=0x7fffffffeb30) at perf.c:556
the command I used is:
perf $ sudo ./perf trace -aS --summary-mode=total --bpf-summary -- sleep 1
[sudo] password for howard:
perf: Segmentation fault
Obtained 14 stack frames.
./perf(dump_stack+0x35) [0x591e26c8a735]
./perf(sighandler_dump_stack+0x2d) [0x591e26c8a7dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x45250) [0x737072045250]
./perf(+0x1543ed) [0x591e26ba43ed]
./perf(+0xff616) [0x591e26b4f616]
./perf(+0xfc3f4) [0x591e26b4c3f4]
./perf(+0x10193b) [0x591e26b5193b]
./perf(cmd_trace+0x205e) [0x591e26b56a9e]
./perf(+0x10b0e0) [0x591e26b5b0e0]
./perf(+0x10b3fb) [0x591e26b5b3fb]
./perf(main+0x2fb) [0x591e26ad63eb]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a3b8) [0x73707202a3b8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x73707202a47b]
./perf(_start+0x25) [0x591e26ad6a35]
Segmentation fault
Thanks,
Howard