Re: [PATCH v13 00/48] arm64: Support for Arm CCA in KVM
From: Suzuki K Poulose
Date: Thu Mar 26 2026 - 07:28:20 EST
Hi Gavin,
On 26/03/2026 00:48, Gavin Shan wrote:
Hi Suzuki,
On 3/25/26 8:16 PM, Suzuki K Poulose wrote:
On 25/03/2026 06:37, Gavin Shan wrote:
On 3/21/26 2:45 AM, Steven Price wrote:
[...]
In upstream TF-A repository [1], I don't see the config option 'RMM_V1_COMPAT'.
would it be something else?
[1] git@xxxxxxxxxx:ARM-software/arm-trusted-firmware.git (branch: master)
suzuki@ewhatever:trusted-firmware-a$ git grep RMM_V1_COMPAT
Makefile: RMM_V1_COMPAT \
Makefile: RMM_V1_COMPAT \
docs/getting_started/build-options.rst:- ``RMM_V1_COMPAT``: Boolean flag to enable support for RMM v1.x compatibility
include/services/rmmd_svc.h:#if RMM_V1_COMPAT
include/services/rmmd_svc.h:#endif /* RMM_V1_COMPAT */
make_helpers/defaults.mk:RMM_V1_COMPAT := 1
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if !RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
suzuki@ewhatever:trusted-firmware-a$ git log --oneline -1
8dae0862c (HEAD, origin/master, origin/integration, origin/HEAD) Merge changes from topic "qti_lemans_evk" into integration
suzuki@ewhatever:trusted-firmware-a$ git remote get-url origin
https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
Thanks for the details. It turned out that I used the wrong TF-A repository. In
the proposed repository, I'm able to see the option 'RMM_V1_COMPAT' and the EL3-RMM
interface compatible issue disappears. However, there are more issues popped up.
I build everything manually where the host is emulated by QEMU instead of shrinkwrap
and FVP model. It's used to work well before. Maybe it's time to switch to shinkwrap
and FVP model since device assignment (DA) isn't supported by an emulated host
by QEMU and shrinkwrap and FVP model seems the only option. I need to learn how
to do that later.
Thanks for the update. Yes, QEMU TF-RMM support is in progress, @Mathieu Poirier is looking into it
There are two issues I can see with the following combinations. Details are provided
like below.
QEMU: https://git.qemu.org/git/ qemu.git (branch: stable-9.2)
TF-RMM: https://git.trustedfirmware.org/TF-RMM/tf- rmm.git (branch: topics/rmm-v2.0-poc)
EDK2: git@xxxxxxxxxx:tianocore/ edk2.git (tag: edk2-stable202411)
TF-A: https://git.trustedfirmware.org/TF-A/trusted-firmware- a.git (branch: master)
HOST: https://git.gitlab.arm.com/linux-arm/linux- cca.git (branch: cca-host/v13)
BUILDROOT: https://github.com/buildroot/ buildroot (branch: master)
KVMTOOL: https://gitlab.arm.com/linux-arm/kvmtool- cca (branch: cca/v11)
GUEST: https://github.com/torvalds/ linux.git (branch: master)
(1) The emulated host is started by the following command lines.
sudo /home/gshan/sandbox/cca/host/qemu/build/qemu-system- aarch64 \
-M virt,virtualization=on,secure=on,gic- version=3,acpi=off \
-cpu max,x-rme=on -m 8G -smp 8 \
-serial mon:stdio -monitor none -nographic - nodefaults \
-bios /home/gshan/sandbox/cca/host/tf-a/ flash.bin \
-kernel /home/gshan/sandbox/cca/host/linux/arch/arm64/boot/ Image \
-initrd /home/gshan/sandbox/cca/host/buildroot/output/images/ rootfs.cpio.xz \
-device pcie-root- port,bus=pcie.0,chassis=1,id=pcie.1 \
-device pcie-root- port,bus=pcie.0,chassis=2,id=pcie.2 \
-device pcie-root- port,bus=pcie.0,chassis=3,id=pcie.3 \
-device pcie-root- port,bus=pcie.0,chassis=4,id=pcie.4 \
-device virtio-9p- device,fsdev=shr0,mount_tag=shr0 \
-fsdev local,security_model=none,path=/home/gshan/sandbox/cca/ guest,id=shr0 \
-netdev tap,id=tap1,script=/etc/qemu-ifup-gshan,downscript=/etc/ qemu-ifdown-gshan \
-device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:f1
(2) Issue-1: TF-RMM complains about the root complex list is invalid. This error is
raised in TF-RMM::setup_root_complex_list() where the error code is still set to
0 (SUCCESS) in this failing case. The TF-RMM initialization is terminated early,
but TF-A still thinks the initialization has been completely done.
INFO: BL31: Initializing RMM
INFO: RMM init start.
RMM EL3 compat memory reservation enabled.
Dynamic VA pool base address: 0xc0000000
Reserved 20 pages. Remaining: 3615 pages
Reserve mem: 20 pages at PA: 0x401f2000 (alignment 0x1000)
Static Low VA initialized. xlat tables allocated: 20 used: 7
Reserved 514 pages. Remaining: 3101 pages
Reserve mem: 514 pages at PA: 0x40206000 (alignment 0x1000)
Dynamic Low VA initialized. xlat tables allocated: 514 used: 514
Invalid: Root Complex list <<<<< ERROR
INFO: RMM init end.
(3) Issue-2: The host kernel gets stuck in rmi_check_version() where SMC_RMI_VERSION
is issued to TF-A, but it can't be forwarded to TF-RMM because its initialization
isn't completely done (issue-1).
[ 37.438253] Unpacking initramfs...
[ 37.563460] kvm [1]: nv: 570 coarse grained trap handlers
[ 37.581139] kvm [1]: nv: 664 fine grained trap handlers
<... system becomes stuck here ...>
So my workaround is to skip fetching root complex list from the EL3-RMM manifest data
in TF-RMM::setup_root_complex_list() since it's not provided for the qemu platform by
^^ This may have to do with the RMM<->TF-A Manifest changes
TF-A. With this workaround, the host can boot up into shell prompt and the guest can
be started by kvmtool.
host$ uname -r
7.0.0-rc1-gavin-gd62aa44b2590
host$ lkvm run --realm -c 2 -m 256 \
-k /mnt/linux/arch/arm64/boot/Image \
-i /mnt/buildroot/output/images/rootfs.cpio.xz
-p earlycon=uart,mmio,0x101000000
Info: # lkvm run -k /mnt/linux/arch/arm64/boot/Image -m 256 -c 2 -- name guest-163
Info: Enabling Guest memfd for confidential guest
Warning: The maximum recommended amount of VCPUs is 1
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
[ 0.000000] Linux version 7.0.0-rc2-gavin-g0031c06807cf (gshan@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-64.el10) #2 SMP PREEMPT Wed Mar 25 20:28:05 EDT 2026
[ 0.000000] KASLR enabled
:
[ 267.578060] Freeing initrd memory: 4728K
[ 267.921865] Warning: unable to open an initial console.
[ 270.327960] Freeing unused kernel memory: 1792K
[ 270.669368] Run /init as init process
Cool, thanks!
Suzuki
Thanks,
Gavin