Re: [PATCH v16 44/45] KVM: arm64: CCA: Require ICH_HCR_EL2.TDIR for realms

[Kohei Enju]

On 08/27 21:45, Kohei Enju wrote:
> On 08/24 15:50, Steven Price wrote:
> > On 10/08/2026 05:58, Kohei Enju wrote:
> > > On 08/03 14:44, Steven Price wrote:
> > >> KVM advertises realm support when the RMM is available, and allows
> > >> userspace to create a VM with KVM_VM_TYPE_ARM_REALM on that basis.
> > >>
> > >> On CPUs that lack ICH_HCR_EL2.TDIR, KVM uses ICH_HCR_EL2.TC for
> > >> normal guests so that ICC_DIR_EL1 is still trapped via the common GICv3
> > >> CPU interface trap. Realms cannot rely on the normal hyp-side trap
> > >> handling for that fallback, so advertising RMI support on such systems
> > >> lets userspace create a realm that cannot safely run.
> > >>
> > >> Require the finalized ARM64_HAS_ICH_HCR_EL2_TDIR capability when
> > >> reporting KVM_CAP_ARM_RMI and when accepting KVM_VM_TYPE_ARM_REALM.
> > >> This leaves normal VM creation unchanged on systems that need the TC
> > >> workaround.
> > > 
> > > Hi Steven,
> > 
> > Hi Kohei,
> > 
> > Sorry for the slow response.
> > 
> > > Thanks for your work on upstreaming CCA.
> > > 
> > > In the v15 discussion [0], you asked whether the system I was testing was a
> > > "hacked up test system" or closer to "production hardware", and I said I would
> > > share more when the time came. I can now say that this is not a hacked-up test
> > > system. At Fujitsu, we have real hardware (FUJITSU-MONAKA) which implements CCA
> > > (FEAT_RME) but does not implement FEAT_GICv3_TDIR. The hardware details are as
> > > follows:
> > 
> > Cool, I suspected that might be the case - it's good to know there's
> > real hardware on it's way.
> > 
> > >   - GICv4.2 compliant implementation
> > >   - Supports FEAT_GICv3, FEAT_GICv3p1, FEAT_GICv4, FEAT_GICv4p1, and FEAT_GICv3_NMI
> > >   - Does not support FEAT_GICv3_LEGACY (deprecated)
> > >   - Does not support FEAT_GICv3_TDIR (ICH_VTR_EL2.TDS == 0)
> > > 
> > > For reference, compared with Arm Neoverse V3, the virtual GIC configuration is
> > > largely equivalent. The only missing non-deprecated architectural feature is
> > > FEAT_GICv3_TDIR.
> > > 
> > > The issue I see is that the CCA KVM code currently does not support a
> > > configuration (non-TDIR/common-trap) that normal KVM already supports. For
> > > normal guests, KVM handles systems without TDIR by using ICH_HCR_EL2.TC and the
> > > existing GICv3 CPU interface emulation path. However, Realm guests currently
> > > fail because the CCA path bypasses that existing emulation path, as Marc also
> > > pointed out in [1].
> > > 
> > > Also, this is not limited to systems that actually lack TDIR. The same failure
> > > can be reproduced on a TDIR-capable system by booting with:
> > >   kvm-arm.vgic_v3_common_trap=1
> > 
> > As Marc says that's a debugging option - handy for those of us who don't
> > have a platform without TDIR to test with.
> > 
> > > So it seems that the current CCA KVM implementation does not yet cover a
> > > configuration that normal KVM already supports today, rather than this being a
> > > limitation of the RMM specification or the underlying hardware.
> > > 
> > > I've included a patch below which reuses the existing GICv3 early emulation
> > > path for Realm sysreg exits. This patch does not add any new vGIC emulation
> > > code, and leaves the existing vGIC emulation code unchanged. So I believe this
> > > is in line with Marc's request in [1]. With this patch, Realm guests can run
> > > when the common CPU interface trap path is enabled.
> > > 
> > > I tested the exact patch both on our real silicon and on QEMU, and
> > > confirmed that all Realm-related tests in kvm-unit-tests-cca passed.
> > > 
> > > I'm not attached to this exact implementation, and I'm happy if the solution is
> > > reworked to better fit into the next revision. 
> > > 
> > > Given that this configuration can be supported by reusing the existing KVM
> > > emulation infrastructure, I think it would be reasonable for CCA to support the
> > > non-TDIR/common-trap configuration rather than requiring ICH_HCR_EL2.TDIR
> > > unconditionally for Realm support. 
> > > 
> > > Supporting this configuration would also allow us to validate the upstream CCA
> > > KVM implementation on real silicon using upstream code paths, and contribute
> > > additional real-hardware testing coverage as the implementation
> > > evolves.
> > > 
> > > I'd be very interested in hearing your thoughts.
> > 
> > So personally I think your patch is a good compromise. It gets the
> > hardware working and I'm keen to enable real-hardware testing. Marc has
> > a very valid point that in terms of performance this could be very bad.
> > Pseudo NMI in particular will be terrible because accesses to GIC
> > registers are used to "emulate" the NMI so the number of traps will be
> > large, and the traps are much more expensive with CCA.
> 
> Yes, in that case ICV_PMR_EL1 would be accessed frequently, and the
> resulting traps would be expensive.
> 
> > 
> > So I'll attempt to incorporate the changes in your patch, but obviously
> > you'll have to decide for yourself whether the performance of the
> > product is suitable.
> 
> Thanks for the clarification.
> 
> I agree with the performance concern, and I'll run some benchmarks on
> our hardware.

Hi Steven and Marc,

Sorry for the late response.

The results confirm your concern. Our initial hackbench measurements showed
considerable performance overhead for the non-TDIR Realm configuration with
pseudo-NMI enabled. Due to restrictions on disclosing performance data for our
hardware, I cannot share the absolute results or further platform details.

However, with pseudo-NMI disabled, performance was comparable to the
TDIR-capable configuration. The difference was at most a few percent across
runs.

There is a real need to run Realm VMs in environments that do not require
pseudo-NMI, and non-TDIR hardware can meet that need. Pseudo-NMI is not enabled
by default in many commonly used distributions, and native NMI support through
FEAT_NMI may also reduce the need for pseudo-NMI in the near future. As a
result, the number of use cases in which the observed performance limitation is
relevant is likely to decrease. Therefore, the pseudo-NMI result does not
necessarily make non-TDIR Realm support impractical for all current or future
use cases.

Note that the initial measurements used pre-production versions of TF-RMM and
TF-A. During our evaluation, we applied preliminary tuning to these firmware
components and have already observed substantial performance improvements.

For the production release of our hardware, we plan to further tune the
software stack, including these firmware components, together with the hardware
configuration. We therefore believe that there remains meaningful room for
further performance improvement, although we cannot yet quantify the final
improvement.

I agree that non-TDIR has a performance limitation for trap-intensive
configurations. However, given the actual customer demand for running CCA
workloads that do not require pseudo-NMI, I believe that supporting Realm VMs
on non-TDIR hardware still has clear practical value. It would also allow us to
accelerate validation of the upstream CCA KVM implementation on real silicon
using upstream code paths and contribute additional real-hardware test coverage
as the implementation evolves.

Thanks,
Kohei

> 
> Thanks,
> Kohei
> 
> > 
> > Thanks,
> > Steve
> >