Re: [PATCH v5 0/6] KVM: arm64: nv: Implement nested stage-2 reverse map (new data structure)

[Wei-Lin Chang]

On Sat, Sep 05, 2026 at 11:49:32PM +0800, Shuai Xue wrote:

[...]

> 
> Hi Wei-Lin,
> 
> Your PARange analysis is what this whole diagnosis rests on -- the
> 48-bit/256x arithmetic is exactly right, and our ftrace data confirms
> it to the chunk: every full-IPA kvm_nested_s2_unmap() in our traces
> walks precisely 262,144 1GB chunks, the number from your table. The
> kvm_ipa_limit=40 experiment also pointed straight at the answer. I'd
> only push back on the last sentence:
> 
> > 
> > With this, I think there aren't other underlying issues, it just is that
> > slow unfortunately..
> 
> 
> I don't think it is "just that slow" -- I think it is a specific
> regression, and your own data localizes it better than the conclusion
> suggests.
> 
> The thing is, kvm_ipa_limit=40 cannot tell *what* each chunk spends
> its ~3.3us on, because the walk is also per-chunk: cutting the chunk
> count by 256x shrinks both terms together, whichever one dominates.
> The experiment that separates them is one where the chunk count does
> not change at all.
> 
> We ran that experiment. Details are in my reply to Marc, but the
> short version:
> 
>   - Before: 877ms per callback, 262,144 chunk calls, 262,144
>     broadcasts. Chunk count and broadcast count are exactly equal
>     (8,476,033 = 8,476,033 over the trace window) -- every chunk
>     ends in a broadcast, mapped or not. Per chunk, ~4.6us of the
>     ~5.1us is inside kvm_tlb_flush_vmid_range(); the walk of an
>     empty chunk is ~0.2us. In an uncontended numad episode, 87% of
>     the 1312ms is flush time and 8% is walk time.
> 
>   - After conditioning the flush on the walk having actually cleared
>     a valid leaf (the candidate fix in my other mail): same 262,144
>     chunks, same walk, ~110ms instrumented. Nothing about the walk
>     changed -- ~3.1us of the per-chunk 3.3us was the broadcast.
> 
>   - And the table really is empty, as Marc says: with the flush
>     count acting as a detector, the first full-IPA unmap of the boot
>     issues 2 flushes (the EL1-boot-period mappings, a few dozen
>     pages in two 1GB chunks); all 35 subsequent ones issue zero.
> 
> The code history agrees. Before 7657ea920c54, the TLBI lived inside
> stage2_put_pte()'s 'if (kvm_pte_valid(ctx->old))' -- an empty walk
> issued zero invalidations, ever. v6.6 hoisted the invalidation out
> to the end of kvm_pgtable_stage2_unmap() to batch it into one range
> TLBI per call, but the condition got dropped on the way. So "since
> v6.6, the cost is proportional to the IPA range, not to the number
> of mappings" -- which is precisely why the empty table pays in full,
> and why I'd frame it as a fixable regression rather than an inherent
> cost.
> 
> None of this subtracts from your PARange finding -- it depends on
> it. The 42s -> <1s result was the 256x chunk scaling; the remaining
> <1s is what the conditional flush removes.

Thanks for the analysis!

I completely agree. My finding was only a small part of the full
picture. I stopped at finding out the real iteration count and didn't
realize 3-5us is a long time for a kvm_pgtable_stage2_unmap() on a empty
table (thanks for the learning opportunity).

So, because there are no relevant tlb entries created in the first
place, the per iteration TLBIs and barriers are mostly doing nothing
useful and cause a 10x slowdown.

Thanks,
Wei-Lin Chang

> 
> Thanks,
> Shuai
> 
>