Re: [PATCH bpf-next v4 0/3] Annotate kfuncs in .BTF_ids section
From: Manu Bretelle
Date: Fri Feb 02 2024 - 18:09:32 EST
On Sun, Jan 28, 2024 at 06:24:05PM -0700, Daniel Xu wrote:
> === Description ===
>
> This is a bpf-treewide change that annotates all kfuncs as such inside
> .BTF_ids. This annotation eventually allows us to automatically generate
> kfunc prototypes from bpftool.
>
> We store this metadata inside a yet-unused flags field inside struct
> btf_id_set8 (thanks Kumar!). pahole will be taught where to look.
>
> More details about the full chain of events are available in commit 3's
> description.
>
> The accompanying pahole and bpftool changes can be viewed
> here on these "frozen" branches [0][1].
>
> [0]: https://github.com/danobi/pahole/tree/kfunc_btf-v3-mailed
> [1]: https://github.com/danobi/linux/tree/kfunc_bpftool-mailed
I hit a similar issue to [0] on master
943b043aeecc ("selftests/bpf: Fix bench runner SIGSEGV")
when cross-compiling on x86_64 (LE) to s390x (BE).
I do have CONFIG_DEBUG_INFO_BTF enable and the issue would not trigger if
I disabled CONFIG_DEBUG_INFO_BTF (and with the fix mentioned in [0]).
What seems to happen is that `tools/resolve_btfids` is ran in the context of the
host endianess and if I printk before the WARN_ON:
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index ef380e546952..a9ed7a1a4936 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -8128,6 +8128,7 @@ int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,
* WARN() for initcall registrations that do not check errors.
*/
if (!(kset->set->flags & BTF_SET8_KFUNCS)) {
+ printk("Flag 0x%08X, expected 0x%08X\n", kset->set->flags, BTF_SET8_KFUNCS);
WARN_ON(!kset->owner);
return -EINVAL;
}
the boot logs would show:
Flag 0x01000000, expected 0x00000001
The issue did not happen prior to
6f3189f38a3e ("bpf: treewide: Annotate BPF kfuncs in BTF")
has only 0 was written before.
It seems [1] will be addressing cross-compilation, but it did not fix it as is
by just applying on top of master, so probably some of the changes will also need
to be ported to `tools/include/linux/btf_ids.h`?
A hacky workaround to cross-compilation I have is to apply:
diff --git a/tools/bpf/resolve_btfids/Makefile b/tools/bpf/resolve_btfids/Makefile
index 4b8079f294f6..b706e7ab066f 100644
--- a/tools/bpf/resolve_btfids/Makefile
+++ b/tools/bpf/resolve_btfids/Makefile
@@ -22,10 +22,10 @@ HOST_OVERRIDES := AR="$(HOSTAR)" CC="$(HOSTCC)" LD="$(HOSTLD)" ARCH="$(HOSTARCH)
CROSS_COMPILE="" EXTRA_CFLAGS="$(HOSTCFLAGS)"
RM ?= rm
-HOSTCC ?= gcc
-HOSTLD ?= ld
-HOSTAR ?= ar
-CROSS_COMPILE =
+HOSTCC = $(CC)
+HOSTLD = $(LD)
+HOSTAR = $(AR)
+#CROSS_COMPILE =
OUTPUT ?= $(srctree)/tools/bpf/resolve_btfids/
@@ -56,16 +56,16 @@ $(OUTPUT) $(OUTPUT)/libsubcmd $(LIBBPF_OUT):
$(SUBCMDOBJ): fixdep FORCE | $(OUTPUT)/libsubcmd
$(Q)$(MAKE) -C $(SUBCMD_SRC) OUTPUT=$(SUBCMD_OUT) \
- DESTDIR=$(SUBCMD_DESTDIR) $(HOST_OVERRIDES) prefix= subdir= \
+ DESTDIR=$(SUBCMD_DESTDIR) prefix= subdir= \
$(abspath $@) install_headers
$(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(LIBBPF_OUT)
$(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) OUTPUT=$(LIBBPF_OUT) \
- DESTDIR=$(LIBBPF_DESTDIR) $(HOST_OVERRIDES) prefix= subdir= \
+ DESTDIR=$(LIBBPF_DESTDIR) prefix= subdir= \
$(abspath $@) install_headers
-LIBELF_FLAGS := $(shell $(HOSTPKG_CONFIG) libelf --cflags 2>/dev/null)
-LIBELF_LIBS := $(shell $(HOSTPKG_CONFIG) libelf --libs 2>/dev/null || echo -lelf)
+LIBELF_FLAGS := $(shell $(PKG_CONFIG) libelf --cflags 2>/dev/null)
+LIBELF_LIBS := $(shell $(PKG_CONFIG) libelf --libs 2>/dev/null || echo -lelf)
HOSTCFLAGS_resolve_btfids += -g \
-I$(srctree)/tools/include \
@@ -84,7 +84,7 @@ $(BINARY_IN): fixdep FORCE prepare | $(OUTPUT)
$(BINARY): $(BPFOBJ) $(SUBCMDOBJ) $(BINARY_IN)
$(call msg,LINK,$@)
- $(Q)$(HOSTCC) $(BINARY_IN) $(KBUILD_HOSTLDFLAGS) -o $@ $(BPFOBJ) $(SUBCMDOBJ) $(LIBS)
+ $(Q)$(CC) $(BINARY_IN) $(KBUILD_HOSTLDFLAGS) -o $@ $(BPFOBJ) $(SUBCMDOBJ) $(LIBS)
clean_objects := $(wildcard $(OUTPUT)/*.o \
$(OUTPUT)/.*.o.cmd \
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index a38a3001527c..5cd193c04448 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -171,7 +171,7 @@ INCLUDE_DIR := $(SCRATCH_DIR)/include
BPFOBJ := $(BUILD_DIR)/libbpf/libbpf.a
ifneq ($(CROSS_COMPILE),)
HOST_BUILD_DIR := $(BUILD_DIR)/host
-HOST_SCRATCH_DIR := $(OUTPUT)/host-tools
+HOST_SCRATCH_DIR := $(SCRATCH_DIR)
HOST_INCLUDE_DIR := $(HOST_SCRATCH_DIR)/include
else
HOST_BUILD_DIR := $(BUILD_DIR)
This causes `resolve_btfids` to be compiled in the target endianess and gets
magically run provided that the hosts has `qemu-s390x-static` and a functional
binfmt_misc [2] on the host, but having this using host architecture per [1]
is likely better.
Here are steps to reproduce the issue on Ubuntu 23.10 and assuming
danobi/vmtest [3] is installed:
XPLATFORM="s390x"
XARCH="s390"
# Set up repo for s390x
cat <<EOF >> /etc/apt/sources.list.d/s390x.list
deb [arch=s390x] http://ports.ubuntu.com/ubuntu-ports mantic main restricted
deb [arch=s390x] http://ports.ubuntu.com/ubuntu-ports mantic-updates main restricted
EOF
sudo dpkg --add-architecture s390x
apt install qemu-system-s390x qemu-user-static g{cc,++}-"${XARCH}-linux-gnu" {libelf-dev,libssl-dev,pkgconf}:s390x
KBUILD_OUTPUT_DIR="/tmp/kbuild-${XPLATFORM}"
mkdir "${KBUILD_OUTPUT_DIR}"
cat tools/testing/selftests/bpf/config{,.vm,.${XPLATFORM}} > ${KBUILD_OUTPUT_DIR}/.config
make ARCH="${XARCH}" CROSS_COMPILE="${XPLATFORM}-linux-gnu-" O="${KBUILD_OUTPUT_DIR}" -j$((4 * $(nproc))) olddefconfig
make ARCH="${XARCH}" CROSS_COMPILE="${XPLATFORM}-linux-gnu-" O="${KBUILD_OUTPUT_DIR}" -j$((4 * $(nproc))) all
# No need for a s390x ubuntu 23.10 rootfs, we only care about booting the kernel
vmtest -k "${KBUILD_OUTPUT_DIR}/arch/s390/boot/bzImage" -a s390x "uname -m" | cat
For the chroot route, see [4].
[0] https://lore.kernel.org/linux-kernel/20240201155339.2b5936be@xxxxxxxxxxxxxxxx/T/
[1] https://lore.kernel.org/bpf/cover.1706717857.git.vmalik@xxxxxxxxxx/
[2] https://en.wikipedia.org/wiki/Binfmt_misc
[3] https://github.com/danobi/vmtest
[4] https://chantra.github.io/bpfcitools/bpf-cross-compile.html
Manu
>
> === Changelog ===
>
> Changes from v3:
> * Rebase to bpf-next and add missing annotation on new kfunc
>
> Changes from v2:
> * Only WARN() for vmlinux kfuncs
>
> Changes from v1:
> * Move WARN_ON() up a call level
> * Also return error when kfunc set is not properly tagged
> * Use BTF_KFUNCS_START/END instead of flags
> * Rename BTF_SET8_KFUNC to BTF_SET8_KFUNCS
>
> Daniel Xu (3):
> bpf: btf: Support flags for BTF_SET8 sets
> bpf: btf: Add BTF_KFUNCS_START/END macro pair
> bpf: treewide: Annotate BPF kfuncs in BTF
>
> Documentation/bpf/kfuncs.rst | 8 +++----
> drivers/hid/bpf/hid_bpf_dispatch.c | 8 +++----
> fs/verity/measure.c | 4 ++--
> include/linux/btf_ids.h | 21 +++++++++++++++----
> kernel/bpf/btf.c | 8 +++++++
> kernel/bpf/cpumask.c | 4 ++--
> kernel/bpf/helpers.c | 8 +++----
> kernel/bpf/map_iter.c | 4 ++--
> kernel/cgroup/rstat.c | 4 ++--
> kernel/trace/bpf_trace.c | 8 +++----
> net/bpf/test_run.c | 8 +++----
> net/core/filter.c | 20 +++++++++---------
> net/core/xdp.c | 4 ++--
> net/ipv4/bpf_tcp_ca.c | 4 ++--
> net/ipv4/fou_bpf.c | 4 ++--
> net/ipv4/tcp_bbr.c | 4 ++--
> net/ipv4/tcp_cubic.c | 4 ++--
> net/ipv4/tcp_dctcp.c | 4 ++--
> net/netfilter/nf_conntrack_bpf.c | 4 ++--
> net/netfilter/nf_nat_bpf.c | 4 ++--
> net/xfrm/xfrm_interface_bpf.c | 4 ++--
> net/xfrm/xfrm_state_bpf.c | 4 ++--
> .../selftests/bpf/bpf_testmod/bpf_testmod.c | 8 +++----
> 23 files changed, 87 insertions(+), 66 deletions(-)
>
> --
> 2.42.1
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel