Re: [PATCH] perf report: Fix arm64 gap between kernel start and module end

From: Arnaldo Carvalho de Melo
Date: Tue Feb 09 2021 - 07:43:01 EST

Em Mon, Mar 30, 2020 at 10:18:10AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Mon, Mar 30, 2020 at 10:11:29AM -0300, Arnaldo Carvalho de Melo escreveu:
> > Em Mon, Mar 30, 2020 at 03:41:11PM +0800, Kemeng Shi escreveu:
> > > diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
> > > index 393b9895c..37cbfa5e9 100644
> > > --- a/tools/perf/arch/arm64/util/Build
> > > +++ b/tools/perf/arch/arm64/util/Build
> > > @@ -2,6 +2,7 @@ libperf-y += header.o
> > > libperf-y += tsc.o
> > > libperf-y += sym-handling.o
> > > libperf-y += kvm-stat.o
> > > +libperf-y += machine.o
> >
> > You made the patch against an old perf codebase, right? This libperf-y
> > above was changed to perf-y here:
> >
> > commit 5ff328836dfde0cef9f28c8b8791a90a36d7a183
> > Author: Jiri Olsa <jolsa@xxxxxxxxxx>
> > Date: Wed Feb 13 13:32:39 2019 +0100
> >
> > perf tools: Rename build libperf to perf
> >
> > ----
> >
> > I'm fixing this up, please check my perf/core branch later to see that
> > all is working as intended.
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git perf/core
>
> Here it is:
>
>
> From 829b915d7d7eeafbe4af76dce19ccbdc743a43c8 Mon Sep 17 00:00:00 2001
> From: Kemeng Shi <shikemeng@xxxxxxxxxx>
> Date: Mon, 30 Mar 2020 15:41:11 +0800
> Subject: [PATCH 1/1] perf symbols: Fix arm64 gap between kernel start and
> module end
>
> During execution of command 'perf report' in my arm64 virtual machine,
> this error message is showed:
>
> failed to process sample
>
> __symbol__inc_addr_samples(860): ENOMEM! sym->name=__this_module,
> start=0x1477100, addr=0x147dbd8, end=0x80002000, func: 0
>
> The error is caused with path:
> cmd_report
> __cmd_report
> perf_session__process_events
> __perf_session__process_events
> ordered_events__flush
> __ordered_events__flush
> oe->deliver (ordered_events__deliver_event)
> perf_session__deliver_event
> machines__deliver_event
> perf_evlist__deliver_sample
> tool->sample (process_sample_event)
> hist_entry_iter__add
> iter->add_entry_cb(hist_iter__report_callback)
> hist_entry__inc_addr_samples
> symbol__inc_addr_samples
> __symbol__inc_addr_samples
> h = annotated_source__histogram(src, evidx) (NULL)
>
> annotated_source__histogram failed is caused with path:
> ...
> hist_entry__inc_addr_samples
> symbol__inc_addr_samples
> symbol__hists
> annotated_source__alloc_histograms
> src->histograms = calloc(nr_hists, sizeof_sym_hist) (failed)
>
> Calloc failed as the symbol__size(sym) is too huge. As show in error
> message: start=0x1477100, end=0x80002000, size of symbol is about 2G.
>
> This is the same problem as 'perf annotate: Fix s390 gap between kernel
> end and module start (b9c0a64901d5bd)'. Perf gets symbol information from
> /proc/kallsyms in __dso__load_kallsyms. A part of symbol in /proc/kallsyms
> from my virtual machine is as follows:
> #cat /proc/kallsyms | sort
> ...
> ffff000001475080 d rpfilter_mt_reg [ip6t_rpfilter]
> ffff000001475100 d $d [ip6t_rpfilter]
> ffff000001475100 d __this_module [ip6t_rpfilter]
> ffff000080080000 t _head
> ffff000080080000 T _text
> ffff000080080040 t pe_header
> ...
>
> Take line 'ffff000001475100 d __this_module [ip6t_rpfilter]' as example.
> The start and end of symbol are both set to ffff000001475100 in
> dso__load_all_kallsyms. Then symbols__fixup_end will set the end of symbol
> to next big address to ffff000001475100 in /proc/kallsyms, ffff000080080000
> in this example. Then sizeof of symbol will be about 2G and cause the
> problem.
>
> The start of module in my machine is
> ffff000000a62000 t $x [dm_mod]
>
> The start of kernel in my machine is
> ffff000080080000 t _head
>
> There is a big gap between end of module and begin of kernel if a samll
> amount of memory is used by module. And the last symbol in module will
> have a large address range as caotaining the big gap.
>
> Give that the module and kernel text segment sequence may change in
> the future, fix this by limiting range of last symbol in module and kernel
> to 4K in arch arm64.
>
> Signed-off-by: Kemeng Shi <shikemeng@xxxxxxxxxx>
> Cc: Alexander Shishkin <alexander.shishkin@xxxxxxxxxxxxxxx>
> Cc: Hu Shiyuan <hushiyuan@xxxxxxxxxx>
> Cc: Ian Rogers <irogers@xxxxxxxxxx>
> Cc: Jiri Olsa <jolsa@xxxxxxxxxx>
> Cc: Mark Rutland <mark.rutland@xxxxxxx>
> Cc: Namhyung Kim <namhyung@xxxxxxxxxx>
> Cc: Nick Desaulniers <ndesaulniers@xxxxxxxxxx>
> Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
> Cc: Thomas Richter <tmricht@xxxxxxxxxxxxx>
> Cc: hewenliang4@xxxxxxxxxx
> Link: http://lore.kernel.org/lkml/33fd24c4-0d5a-9d93-9b62-dffa97c992ca@xxxxxxxxxx
> [ refreshed the patch on current codebase ]
> Signed-off-by: Arnaldo Carvalho de Melo <acme@xxxxxxxxxx>
> ---
> tools/perf/arch/arm64/util/Build | 1 +
> tools/perf/arch/arm64/util/machine.c | 26 ++++++++++++++++++++++++++
> 2 files changed, 27 insertions(+)
> create mode 100644 tools/perf/arch/arm64/util/machine.c
>
> diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
> index 789956f76d85..5c13438c7bd4 100644
> --- a/tools/perf/arch/arm64/util/Build
> +++ b/tools/perf/arch/arm64/util/Build
> @@ -1,4 +1,5 @@
> perf-y += header.o
> +perf-y += machine.o
> perf-y += perf_regs.o
> perf-$(CONFIG_DWARF) += dwarf-regs.o
> perf-$(CONFIG_LOCAL_LIBUNWIND) += unwind-libunwind.o
> diff --git a/tools/perf/arch/arm64/util/machine.c b/tools/perf/arch/arm64/util/machine.c
> new file mode 100644
> index 000000000000..94745131e89a
> --- /dev/null
> +++ b/tools/perf/arch/arm64/util/machine.c
> @@ -0,0 +1,26 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#include <stdio.h>
> +#include "debug.h"
> +#include "symbol.h"
> +
> +/* On arm64, kernel text segment start at high memory address,
> + * for example 0xffff 0000 8xxx xxxx. Modules start at a low memory
> + * address, like 0xffff 0000 00ax xxxx. When only samll amount of
> + * memory is used by modules, gap between end of module's text segment
> + * and start of kernel text segment may be reach 2G.
> + * Therefore do not fill this gap and do not assign it to the kernel dso map.
> + */
> +
> +#define SYMBOL_LIMIT (1 << 12) /* 4K */
> +
> +void arch__symbols__fixup_end(struct symbol *p, struct symbol *c)
> +{
> + if ((strchr(p->name, '[') && strchr(c->name, '[') == NULL) ||
> + (strchr(p->name, '[') == NULL && strchr(c->name, '[')))
> + /* Limit range of last symbol in module and kernel */
> + p->end += SYMBOL_LIMIT;
> + else
> + p->end = c->start;
> + pr_debug4("%s sym:%s end:%#lx\n", __func__, p->name, p->end);
> +}

This breaks the build on powerpc 32-bit, I'm fixing it using the proper "%#" PRIx64 " style.

- Arnaldo


72 13.69 ubuntu:18.04-x-powerpc : FAIL powerpc-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
arch/powerpc/util/machine.c: In function 'arch__symbols__fixup_end':
arch/powerpc/util/machine.c:23:12: error: format '%lx' expects argument of type 'long unsigned int', but argument 6 has type 'u64 {aka long long unsigned int}' [-Werror=format=]
pr_debug4("%s sym:%s end:%#lx\n", __func__, p->name, p->end);
^
/git/linux/tools/perf/util/debug.h:18:21: note: in definition of macro 'pr_fmt'
#define pr_fmt(fmt) fmt
^~~
/git/linux/tools/perf/util/debug.h:33:29: note: in expansion of macro 'pr_debugN'
#define pr_debug4(fmt, ...) pr_debugN(4, pr_fmt(fmt), ##__VA_ARGS__)
^~~~~~~~~
/git/linux/tools/perf/util/debug.h:33:42: note: in expansion of macro 'pr_fmt'
#define pr_debug4(fmt, ...) pr_debugN(4, pr_fmt(fmt), ##__VA_ARGS__)
^~~~~~
arch/powerpc/util/machine.c:23:2: note: in expansion of macro 'pr_debug4'
pr_debug4("%s sym:%s end:%#lx\n", __func__, p->name, p->end);
^~~~~~~~~
cc1: all warnings being treated as errors
/git/linux/tools/build/Makefile.build:139: recipe for target 'util' failed
make[5]: *** [util] Error 2
/git/linux/tools/build/Makefile.build:139: recipe for target 'powerpc' failed
make[4]: *** [powerpc] Error 2
/git/linux/tools/build/Makefile.build:139: recipe for target 'arch' failed
make[3]: *** [arch] Error 2
73 30.47 ubuntu:18.04-x-powerpc64 : Ok powerpc64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0