[PATCH v1 0/5] KVM: riscv: Age G-stage PTEs locklessly

[SeungJu Cheon]

Aging G-stage PTEs currently runs with mmu_lock held for write, taken
by the common MMU notifier code. When MGLRU or kswapd ages a large
range, every vCPU taking a G-stage fault blocks on the lock, and the
dirty-logging read-side fast path blocks behind aging as well.

This series lets RISC-V select KVM_MMU_LOCKLESS_AGING, so G-stage
aging can run without mmu_lock while protecting page-table walks with
RCU.

  1. Age all G-stage PTEs in a GFN range
     Bug fix, posted separately [1] and included unchanged so the
     series applies on riscv_kvm_queue. Drop it if already applied.

  2. Read G-stage PTEs once when walking page tables
     A lockless walker must derive presence, leaf-ness and the child
     address from a single snapshot of each entry.

  3. Write-protect G-stage PTEs atomically
     Aging will clear the Accessed bit without mmu_lock; the plain
     read-modify-write in GSTAGE_OP_WP could clobber that update.

  4. Free G-stage page tables after an RCU grace period
     Table pages and the root outlive any RCU reader. Root teardown
     keeps pgd_levels intact so a walker holding the old root sees
     consistent metadata.

  5. Age G-stage PTEs locklessly
     Select KVM_MMU_LOCKLESS_AGING and walk under rcu_read_lock().

This builds on Jinyu Tang's rwlock and cmpxchg helpers already in
riscv_kvm_queue. Wang Yechao's huge page recovery series [2] adds a
second page-table free site in kvm_riscv_gstage_recover_huge();
whichever lands second will need to switch that put_page() to
gstage_free_page_table(). I can rebase the series as needed.

A possible follow-up is to run clear-dirty-log and the general fault
path under the read side of mmu_lock, as arm64 and x86 do. Patch 3
provides the atomic write-protection needed for the former.

Testing
-------
QEMU virt, -cpu rv64,h=true (Svadu), 4 vCPUs, TCG. The host kernel
was built with KASAN, PROVE_LOCKING, PROVE_RCU, LOCK_STAT and LRU_GEN.

After each patch:
  - dirty_log_perf_test -v 2 -b 256M -i 3
  - access_tracking_perf_test -v 2 -b 256M -s anonymous_thp
    (2M THP disabled and 64K mTHP enabled)
  - 20 VM create/destroy iterations

With MGLRU aging forced on the VM's cgroup every 50-100ms:
  - dirty_log_perf_test -v 3 -b 256M -i 3
  - 30 VM create/destroy iterations

With CONFIG_KVM=m, 5 module load/unload cycles, each with a VM created
and destroyed immediately before module unload, to exercise module
teardown after queuing RCU frees.

No WARN, KASAN, lockdep or RCU reports were observed. The kvm_age_hva
tracepoint recorded approximately 65k events per aging pass.

Performance results from dirty_log_perf_test -v 3 -b 256M -i 3, with
lock_stat enabled for mmu_lock and 86 aging passes:

                           no aging          aging
                         before  after     before   after
  guest dirty time       12.5s   12.4s     18.6s    16.7s
  mmu_lock W wait total  0.15s   0.17s     44.4s    0.15s
  mmu_lock W contentions  35k     35k      1.0M     30k
  mmu_lock R wait total   0       0        7.5s     0

Before the series, 93% of write-lock waiters were from
kvm_mmu_notifier_clear_young(), and the dirty-log read-side fast path
waited for mmu_lock 352k times. After the series, aging does not
acquire mmu_lock.

Absolute numbers are inflated by TCG; only the relative change is
meaningful. Under TCG, the atomic write-protect change in patch 3
increased the clear-dirty-log time from approximately 16ms to 30ms,
as TCG serialises AMO instructions; hardware numbers would be welcome.

[1] https://lore.kernel.org/all/20260918072622.284188-1-suunj1331@xxxxxxxxx/
[2] https://lore.kernel.org/kvm-riscv/20260914055936.3672758-1-wang.yechao255@xxxxxxxxxx/

SeungJu Cheon (5):
  KVM: riscv: Age all G-stage PTEs in a GFN range
  KVM: riscv: Read G-stage PTEs once when walking page tables
  KVM: riscv: Write-protect G-stage PTEs atomically
  KVM: riscv: Free G-stage page tables after an RCU grace period
  KVM: riscv: Age G-stage PTEs locklessly

 arch/riscv/include/asm/kvm_gstage.h |   4 +-
 arch/riscv/kvm/Kconfig              |   1 +
 arch/riscv/kvm/gstage.c             | 111 +++++++++++++++++++++-------
 arch/riscv/kvm/main.c               |   3 +
 arch/riscv/kvm/mmu.c                |  56 +++++++-------
 5 files changed, 117 insertions(+), 58 deletions(-)


base-commit: 41e81f7e3ef96594fb840445343c0ee7723aa550
-- 
2.52.0