Re: perf 6.9-1 (archlinux) crashes during recording of cycles + raw_syscalls
From: Namhyung Kim
Date: Fri Jun 07 2024 - 14:26:26 EST
On Thu, Jun 06, 2024 at 04:17:43PM -0700, Ian Rogers wrote:
> On Thu, Jun 6, 2024 at 3:20 PM Namhyung Kim <namhyung@xxxxxxxxxx> wrote:
> >
> > On Tue, Jun 04, 2024 at 04:02:08PM -0300, Arnaldo Carvalho de Melo wrote:
> > > On Tue, Jun 04, 2024 at 11:48:09AM -0700, Ian Rogers wrote:
> > > > On Tue, Jun 4, 2024 at 7:12 AM Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:
> > > > > Can you please try with the attached and perhaps provide your Tested-by?
> > >
> > > > > From ab355e2c6b4cf641a9fff7af38059cf69ac712d5 Mon Sep 17 00:00:00 2001
> > > > > From: Arnaldo Carvalho de Melo <acme@xxxxxxxxxx>
> > > > > Date: Tue, 4 Jun 2024 11:00:22 -0300
> > > > > Subject: [PATCH 1/1] Revert "perf record: Reduce memory for recording
> > > > > PERF_RECORD_LOST_SAMPLES event"
> > >
> > > > > This reverts commit 7d1405c71df21f6c394b8a885aa8a133f749fa22.
> > >
> > > > I think we should try to fight back reverts when possible. Reverts are
> > > > removing something somebody poured time and attention into. When a
> > >
> > > While in the development phase, yeah, but when we find a regression and
> > > the revert makes it go away, that is the way to go.
> > >
> > > The person who poured time on the development gets notified and can
> > > decide if/when to try again.
> > >
> > > Millian had to pour time to figure out why something stopped working,
> > > was kind enough to provide the output from multiple tools to help in
> > > fixing the problem and I had to do the bisect to figure out when the
> > > problem happened and to check if reverting it we would have the tool
> > > working again.
> > >
> > > If we try to fix this for v6.10 we may end up adding yet another bug, so
> > > the safe thing to do at this point is to do the revert.
> > >
> > > We can try improving this once again for v6.11.
> >
> > I think I found a couple of problems with this issue. :(
> >
> > 1. perf_session__set_id_hdr_size() uses the first evsel in the session
> > But I think it should pick the tracking event. I guess we assume
> > all events have the same set of sample_type wrt the sample_id_all
> > but I'm not sure if it's correct.
> >
> > 2. With --call-graph dwarf, it seems to set unrelated sample type bits
> > in the attr like ADDR and DATA_SRC.
> >
> > 3. For tracepoint events, evsel__newtp_idx() sets a couple of sample
> > type regardless of the configuration. This includes RAW, TIME and
> > CPU. This one changes the format of the id headers.
> >
> > 4. PERF_RECORD_LOST_SAMPLES is for the sampling event, so it should
> > use the event's sample_type. But the event parsing looks up the
> > event using evlist->is_pos which is set for the first event.
> >
> > 5. I think we can remove some sample type (i.e. TID and CPU) from the
> > tracking event in most cases. ID(ENTIFIER) will be used for LOST_
> > SAMPLES and TIME is needed anyway. TID is might be used for SWITCH
> > but others already contain necessary information in the type. I
> > wish we could add id field to PERF_RECORD_LOST_SAMPLES and tid/pid
> > to PERF_RECORD_SWITCH.
>
> Right, this is good. To clear up the immediate error we just need to
> increase the memory allocation size by 48 bytes to account for the
> sample ID being written. Here is a change doing that and removing the
> memory allocation altogether:
Looks ok. I think we can have this as a quick fix and I can work on the
above issues separately. Can you please send a formal patch?
> ```
> diff --git a/tools/lib/perf/include/perf/event.h
> b/tools/lib/perf/include/perf/event.h
> index ae64090184d3..a2dfaff26fb7 100644
> --- a/tools/lib/perf/include/perf/event.h
> +++ b/tools/lib/perf/include/perf/event.h
> @@ -77,6 +77,8 @@ struct perf_record_lost_samples {
> __u64 lost;
> };
>
> +#define PERF_RECORD_MAX_LOST_SAMPLE_AND_ID_SIZE \
> + (sizeof(struct perf_record_lost_samples) + 6 * sizeof(__u64))
For possible other uses, I'd suggest to add
#define MAX_ID_HDR_ENTRIES 6
Thanks,
Namhyung
> /*
> * PERF_FORMAT_ENABLED | PERF_FORMAT_RUNNING | PERF_FORMAT_ID | PERF_FORMAT_LOST
> */
> diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
> index 66a3de8ac661..1615a1723fb9 100644
> --- a/tools/perf/builtin-record.c
> +++ b/tools/perf/builtin-record.c
> @@ -1926,7 +1926,10 @@ static void __record__save_lost_samples(struct
> record *rec, struct evsel *evs
> el,
> static void record__read_lost_samples(struct record *rec)
> {
> struct perf_session *session = rec->session;
> - struct perf_record_lost_samples *lost = NULL;
> + union {
> + struct perf_record_lost_samples lost;
> + char
> lost_and_sample_id[PERF_RECORD_MAX_LOST_SAMPLE_AND_ID_SIZE];
> + } lost;
> struct evsel *evsel;
>
> /* there was an error during record__open */
> @@ -1951,20 +1954,13 @@ static void record__read_lost_samples(struct
> record *rec)
>
> if (perf_evsel__read(&evsel->core, x,
> y, &count) < 0) {
> pr_debug("read LOST count failed\n");
> - goto out;
> + return;
> }
>
> if (count.lost) {
> - if (!lost) {
> - lost = zalloc(sizeof(*lost) +
> -
> session->machines.host.id_hdr_size);
> - if (!lost) {
> -
> pr_debug("Memory allocation failed\n");
> - return;
> - }
> - lost->header.type =
> PERF_RECORD_LOST_SAMPLES;
> - }
> -
> __record__save_lost_samples(rec, evsel, lost,
> + memset(&lost, 0, sizeof(lost));
> + lost.lost.header.type =
> PERF_RECORD_LOST_SAMPLES;
> +
> __record__save_lost_samples(rec, evsel, &lost.lost,
> x,
> y, count.lost, 0);
> }
> }
> @@ -1972,21 +1968,12 @@ static void record__read_lost_samples(struct
> record *rec)
>
> lost_count = perf_bpf_filter__lost_count(evsel);
> if (lost_count) {
> - if (!lost) {
> - lost = zalloc(sizeof(*lost) +
> -
> session->machines.host.id_hdr_size);
> - if (!lost) {
> - pr_debug("Memory allocation failed\n");
> - return;
> - }
> - lost->header.type = PERF_RECORD_LOST_SAMPLES;
> - }
> - __record__save_lost_samples(rec, evsel, lost,
> 0, 0, lost_count,
> + memset(&lost, 0, sizeof(lost));
> + lost.lost.header.type = PERF_RECORD_LOST_SAMPLES;
> + __record__save_lost_samples(rec, evsel,
> &lost.lost, 0, 0, lost_count,
>
> PERF_RECORD_MISC_LOST_SAMPLES_BPF);
> }
> }
> -out:
> - free(lost);
> }
>
> static volatile sig_atomic_t workload_exec_errno;
> ```
>
> Thanks,
> Ian
>
> > Thanks,
> > Namhyung
> >
> > >
> > > > regression has occurred then I think we should add the regression case
> > > > as a test.
> > >
> > > Sure, I thought about that as well, will try and have one shell test
> > > with that, referring to this case, for v6.11.
> > >
> > > - Arnaldo