Re: [PATCH 00/17] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting

[KunWu Chan]

On Thu, Sep 24, 2026 at 5:56 AM David Woodhouse <dwmw2@xxxxxxxxxxxxx> wrote:
>
> On Thu, 2026-09-24 at 00:26 +0800, KunWu Chan wrote:
> > The parallel interrupt delivery case sounds particularly interesting.
> > Please don't go to any trouble if the old results are not available.
>
> The old results were on the original RCU-based (not SRCU) version, so
> I redid them with the latest, on the 192-way PREEMPT_RT GNR box.
>
> It's a bit of a microbenchmark: N vCPUs spinning in guest mode, with N
> more sender threads, each delivering interrupts as fast as it can to
> its own paired vCPU thread. The vCPU isn't even acknowledging the
> interrupt, but the locked btsl will still be dirtying the cache lines
> of the shared info evtchn_pending bitmap even to find that the bit is
> already set (qv).
>
> The rwlock version gets up to 8-9M deliveries per second and then
> starts to regress past about 32 parallel vCPUs. While SRCU starts off
> faster, and gets more so.
>
> Deliveries/sec, spread layout:
>   ┌───────┬──────────┬─────────────┬───────┐
>   │ vCPUs │  rwlock  │ atomic SRCU │ ratio │
>   ├───────┼──────────┼─────────────┼───────┤
>   │ 1     │ 1.02M    │ 1.13M       │ 1.1×  │
>   ├───────┼──────────┼─────────────┼───────┤
>   │ 2     │ 1.87M    │ 2.27M       │ 1.2×  │
>   ├───────┼──────────┼─────────────┼───────┤
>   │ 4     │ 3.59M    │ 4.57M       │ 1.3×  │
>   ├───────┼──────────┼─────────────┼───────┤
>   │ 8     │ 7.06M    │ 9.13M       │ 1.3×  │
>   ├───────┼──────────┼─────────────┼───────┤
>   │ 16    │ 8.75M    │ 16.8M       │ 1.9×  │
>   ├───────┼──────────┼─────────────┼───────┤
>   │ 32    │ 8.52M    │ 33.3M       │ 3.9×  │
>   ├───────┼──────────┼─────────────┼───────┤
>   │ 64    │ 6.75M    │ 63.7M       │ 9.4×  │
>   └───────┴──────────┴─────────────┴───────┘
>
> This is with the port numbers of each of the N CPUs deliberately spread
> out into different cache lines (which arguably we ought to do for
> production guests, but we don't). If I let them all pack with
> sequential port# allocation, it makes no difference to rwlock but the
> SRCU variant hits a scaling cliff around N=16 too (17.2M @32, 14.0M
> @64). But hey, as I said: microbenchmark.

Resend.

Hi David,

Thanks for taking the time to rerun this. This is exactly the kind of
data I was hoping to get, and the scaling at 32 and 64 vCPUs is
particularly useful.

The cache-line layout comparison is also a useful data point. I'll keep
the microbenchmark limitation in mind when using these results.

Thanks again for the extra work.

Thanks,
Kunwu