Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)

[Marc Zyngier]

On Fri, 04 Sep 2026 08:01:24 +0100,
Shuai Xue <xueshuai@xxxxxxxxxxxxxxxxx> wrote:
> 
> 
> 
> On 9/3/26 9:28 PM, Wei-Lin Chang wrote:
> > On Thu, Sep 03, 2026 at 08:43:35AM +0100, Marc Zyngier wrote:
> >> On Wed, 02 Sep 2026 17:35:00 +0100,
> >> Wang Han <wanghan@xxxxxxxxxxxxxxxxx> wrote:
> >>> 
> >>> Hi Wei-Lin,
> >>> 
> >>> I tested this series on a Yitian 710 system with an ARM Neoverse-N2 CPU
> >>> (128 CPUs, 2 NUMA nodes).
> >>> 
> >>> Test environment
> >>> ----------------
> >>> 
> >>>    L0 kernel: Linux v7.2-rc6
> >>>    L1 guest: Ubuntu 26.04 LTS, kernel 7.0.0-27-generic (aarch64)
> >>>    QEMU: 10.2.3
> >>> 
> >>> L0 NUMA balancing was enabled (`/proc/sys/kernel/numa_balancing=1`).
> >>> The host was booted with `kvm_arm.mode=nested`.
> >>> 
> >>> This series fixes a functional hang that is exposed when NUMA balancing is
> >>> enabled.  The previous nested stage-2 unmap path is too slow for this
> >>> workload, making the performance problem user-visible: NUMA balancing can
> >>> leave the L1 guest unable to make progress and eventually hang during boot.
> >>> 
> >>> The L1 was started with 8 vCPUs and 32 GiB of RAM using:
> >>> 
> >>>    qemu-system-aarch64 -smp 8 -m 32G \
> >>>      -machine virt,accel=kvm,gic-version=3,virtualization=on \
> >>>      -cpu host -nographic -enable-kvm \
> >>>      -drive if=pflash,format=raw,readonly=on,file=pflash0_bak.img \
> >>>      -drive if=pflash,format=raw,file=pflash1_bak.img \
> >>>      -drive file=./ubuntu-vm.qcow2,format=qcow2,if=virtio,cache=none,aio=native \
> >>>      -nic user,model=virtio-net-pci,hostfwd=tcp::11234-:22 \
> >>>      -serial mon:stdio
> >>> 
> >> 
> >> Puzzling. If you are only running an L1 in VHE mode, there is no
> >> shadow S2, and therefore nothing to unmap. For shadow S2s to be built
> >> and affect the MMU notifiers, you need to run an L2.
> > 
> > I was thinking the same at first, but realized even with L1 in VHE mode
> > there is a small period of time where L1 runs in its EL1 during boot, so
> > one nested MMU will become valid for each vCPU. That causes
> > kvm_nested_s2_unmap() to iterate through the entire IPA space 8 times
> > (-smp 8).
> > 
> > What I am curious about is whether one single notifier unmap is enough
> > to hang L1, or were there multiple notifier unmaps.
> > 
> > QEMU with -machine virt uses 40 IPA bits only, unmapping that takes:
> > 1024  (4KB pages,  unmapping 1GB per iteration)
> > 32768 (16KB pages, unmapping 32MB per iteration)
> > 2048  (64KB pages, unmapping 512MB per iteration)
> > iterations for each page size. There aren't many mappings in each
> > iteration too. Does this really take that long on real hardware (even if
> > this must be done 8 times)?
> > 
> > Thanks,
> > Wei-Lin Chang
> > 
> >> 
> >> So what are your actual test conditions?
> >> 
> >> 	M.
> >> 
> 
> Hi, Wei-Lin and Marc,
> 
> I was able to reproduce this issue and capture ftrace evidence that confirms
> the root cause. Below is the analysis, trace log, and timing data.

[...]

> Each set_migration_pte line is a single-page NUMA migration. Yet each
> migration triggers one full kvm_nested_s2_unmap() that takes 877 ms.

And why is it taking so long? It should be *empty* after the first
iteration.

[...]

> ## Conclusion
> 
> The root cause is confirmed: kvm_nested_s2_unmap() performs a full IPA space
> unmap in the MMU notifier path instead of unmapping only the affected
> GPA/CPAI range. The interval-tree-based precise range unmap approach is the
> right fix.

No. This just indicates that this is papering over a bigger problem,
and your AI is jumping to conclusions.

> Please consider applying the patch that replaces the full unmap with
> kvm_nested_unmap_cipa_range() to avoid scanning the entire nested stage-2
> page table on every NUMA migration.

Not until we get to the bottom of this issue.

	M.

-- 
Jazz isn't dead. It just smells funny.