Re: odd endianness toolchains for crosstool

From: Arnd Bergmann
Date: Mon Apr 25 2022 - 10:55:55 EST

On Mon, Apr 25, 2022 at 1:43 PM Jason A. Donenfeld <Jason@xxxxxxxxx> wrote:
> On Mon, Apr 25, 2022 at 10:46:45AM +0200, Arnd Bergmann wrote:
> > The situation on my end is that I'm planning to migrate my main workstation
> > (which I'm building the compilers on) to an arm64 machine soon, and
> > will then need to set it all up again. I don't really want to change much before
> > then to avoid changing things twice.
> Ahh, okay, so probably crosstool won't be viable for me for a while. Are
> your existing scripts fairly reproducible and easy? I suppose I could
> just build my own if I can't find another project supplying light
> compilers.

The scripts are fairly solid, but the original git tree is no longer available
and my version has a couple of local changes with a bit of a dirty history
from adding support for cross-compiling the compilers themselves to
do the canadian-cross arm64 and ppc64le hosted ones.

There is another fork of the same tree on

The main issue with building distributable binaries is to actually build
them on an older rootfs to avoid linking against a newer glibc, and
to ensure the dependencies for gcc are statically linked. Without that,
the output is too distro specific.

> > I've added Nick to Cc, as he's experimenting with a clang based toolchain
> > that we can put on along with the gcc toolchains, and that
> > would probably include a musl based sysroot roughly the same set of
> > architectures that you are testing on already. Possibly we could reuse the
> > same user space between clang and gcc.
> I personally have no use for compilers with user spaces. My test harness
> builds musl as part of it. It's really the quickest part of the entire
> harness to build. I also probably won't switch things over to clang;
> Google has the resources to do that themselves. Basically all I need is
> the boring nolibc compilers for a few extra platforms, and for the ppc
> one to build with the mentioned flags.

I suppose the only thing you are missing is libgcc (or libcompiler-rt)
for those platforms. I had a closer look into what is or can be included
here, and I see that my builds include multiple versions on some of
the architectures, but not on others:


So on powerpc, there are already both big-endian and
little-endian binaries, but arm and mips only have one of the
two. I asked our local compiler experts, and they suggested
that one can add further multiarch-configs like the one
in gcc/config/arm/t-aprofile to allow building for a different
subset of the hundreds of possible configurations.

the t-aprofile builds libgcc for a couple of combinations of
cpu architecture level and FPU ABIs, but they are all the
same endianess. gcc/config/rs6000/t-linux64lebe is the
corresponding file for powerpc that enables all combinations
of 32/64 and be/le.

> > I've also looked at other projects that do qemu based testing, everyone
> > seems to be missing one or two architectures out of a common set,
> > is where I keep my data.
> If the compilers are there, and they can build a working musl, and QEMU
> will boot it, and I can work out a minimal kernel .config that doesn't
> take a long time to compile, then it'll get included in the CI. So in
> theory, I should be able to expand the portfolio of architectures I'm
> using.

Adding riscv and s390 should indeed be fairly simple to add,
and you can probably just take a look at what ktest does for them.

You have a good point about the minimal kernel config, which makes
sense for testing a single thing, but of course others generally want
to test a 'defconfig' build that is closer to what users would actually

> > building and running the most common subset of these in one place
> > in the kernel tree where at least wireguard, kunit and tuxrun can
> > share the setup for qemu.
> I have little interest in that kind of abstraction unfortunately.

Ok, fair enough.