Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)

[Marc Zyngier]

On Fri, 04 Sep 2026 23:42:15 +0100,
Wei-Lin Chang <weilin.chang@xxxxxxx> wrote:
> 
> On Fri, Sep 04, 2026 at 08:49:14AM +0100, Marc Zyngier wrote:
> 
> [...]
> 
> > > > >
> > > > > The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> > > > > 
> > > > >   qemu-system-aarch64 -smp 8 -m 32G \
> > > > >     -machine virt,accel=kvm,gic-version=3,virtualization=on \
> > > > >     -cpu host -nographic -enable-kvm \
> > > > >     -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> > > > >     -drive if=pflash,format=raw,file=pflash1_bak.img \
> > > > >     -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> > > > >     -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> > > > >     -serial mon:stdio
> > > > > 
> > > > 
> > > > Puzzling. If you are only running an L1 in VHE mode, there is no
> > > > shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> > > > and affect the MMU notifiers, you need to run an L2.
> > > 
> > > I was thinking the same at first, but realized even with L1 in VHE mode
> > > there is a small period of time where L1 runs in its EL1 during boot, so
> > > one nested MMU will become valid for each vCPU. That causes
> > > kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> > > (-smp 8).
> > 
> > It should be one nested MMU for the whole VM, not one per vcpu.
> > That's assuming they share the same VMID+VTCR.
> > 
> > > What I am curious about is whether one single notifier unmap is enough
> > > to hang L1, or were there multiple notifier unmaps.
> > > 
> > > QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> > > 1024  (4KB pages,  unmapping 1GB per iteration)
> > > 32768 (16KB pages, unmapping 32MB per iteration)
> > > 2048  (64KB pages, unmapping 512MB per iteration)
> > > iterations for each page size. There aren't many mappings in each
> > > iteration too. Does this really take that long on real hardware (even if
> > > this must be done 8 times)?
> > 
> > This should be close to being at zero cost, so something else is
> > amiss.
> > 
> 
> Guess what? I am completely wrong once again.
> 
> The complete range of the nested mmus isn't about the value
> supplied by KVM_VM_TYPE_ARM_IPA_SIZE. Instead it is the PARange exposed
> to the guest. That's because the guest hypervisor is allowed to expose
> whatever memory map it likes to its nested guests, as long as it is
> within the guest's PARange. This is explained clearly in
> init_nested_s2_mmu().
> 
> Now the PARange exposed to guests should be much larger than 40bits. On
> the neoverse-n2 it should be 48bits (256TB), therefore the iterations
> required for each unmapping would be 256x:
> 
> 262144  (4KB pages,  unmapping 1GB per iteration)
> 8388608 (16KB pages, unmapping 32MB per iteration)
> 524288  (64KB pages, unmapping 512MB per iteration)
> 
> I think these numbers match the hang time Shuai have shown.
>
> Just to demonstrate the difference I have tried manually limiting
> kvm_ipa_limit to 40, and tested with software emulation. Before the
> manual change a kvm_nested_s2_unmap() finished after 42 seconds, after
> limiting kvm_ipa_limit kvm_nested_s2_unmap() can finish in less than a
> second.
>
> With this, I think there aren't other underlying issues, it just is that
> slow unfortunately..

I don't buy this. There should be very little mapped in the shadow S2,
a few dozen pages at most. As you noticed, there is very little the
kernel executes at EL1 before upgrading to EL2. Invalidating this
should be cheap.  Furthermore, once invalidated after the last vcpu
has booted, the cost should be exactly zero.

It seems that these two assumptions are not holding up, and I wonder
why.

	M.

-- 
Jazz isn't dead. It just smells funny.