[PATCH 11/17] KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages

[David Woodhouse]

From: Fred Griffoul <fgriffo@xxxxxxxxxxxx>

Replace kvm_host_map usage with gfn_to_pfn_cache for the L1 pages whose
physical addresses are handed to the CPU in vmcs02 for direct use while
running L2: the APIC-access page, the virtual-APIC page, and the posted
interrupt descriptor. This eliminates the map/unmap (with unmanaged
memory: memremap/memunmap) cycle on every L2 VM-entry/exit.

The caches are initialized with kvm_gpc_init_for_vcpu(), naming
KVM_REQ_GET_NESTED_STATE_PAGES as the service request, and their pfns
are pinned for guest use (GPC_GUEST_USING) when written into vmcs02 by
nested_get_vmcs12_pages(). If an MMU notifier invalidation hits a
pinned cache, the pfncache core posts that request, forces the vCPU
out of guest mode, and waits for it to leave; the request handler
re-runs nested_get_vmcs12_pages(), whose refresh cannot complete until
the invalidation ends, and rewrites the vmcs02 fields before the next
entry. The pins are dropped at nested VM-exit (including the
vmentry_fail_vmexit path, which does not pass through
__nested_vmx_vmexit()) and the caches deactivated when VMX operation
ends.

Note that no validity check is added to the guest-entry path: the
request substitutes for it. The invariant is that a stale physical
address latched in vmcs02 always implies a pending request, which the
existing kvm_request_pending() check catches after vcpu->mode is set
to IN_GUEST_MODE. It is maintained from both directions:

 - Every path which consumes a pin (the notifier walk, or a
   refresh/deactivate from any context, including the owning vCPU
   refreshing its own cache from a host-side reader) posts the
   service request.

 - The request handler cannot lose it: vcpu_enter_guest() consumes
   the request bit before calling in, so ANY failure of
   vmx_get_nested_state_pages() — including early exits which touch
   no cache at all, such as a failed load_pdptrs() — re-posts the
   request before exiting to userspace. A persistently failing
   configuration bounces to userspace on every KVM_RUN rather than
   ever entering L2 with a stale address. (The old kvm_host_map code
   could tolerate losing this race because the map held a page
   reference; the pfncache deliberately holds none.)

This is what closes the historical KVM_GUEST_USES_PFN gaps¹: the kick
is guaranteed to be acted upon before re-entry, without polling cache
validity on every entry.

Host-side (khva) readers of the virtual-APIC and PI descriptor pages
do not need the pin, which exists only for the benefit of the
physical addresses latched in vmcs02.
vmx_complete_nested_posted_interrupt() runs in a sleepable context
and uses the SRCU-protected check/refresh protocol.
vmx_has_nested_events() is reachable from the kvm_vcpu_block() loop
under set_current_state(TASK_INTERRUPTIBLE), where a refresh (mutex,
GUP, synchronize_srcu) must not sleep: it instead uses a non-sleeping
try-lock and reports "event pending" when a cache has been
invalidated. The resulting bounce through vcpu_run services the
pending request in a sleepable context and re-evaluates events for
real; a spurious wakeup is safe, whereas returning false could miss
a wakeup (the service request is posted with KVM_REQUEST_NO_WAKEUP).
To keep an unbackable page from turning that conservative answer into
a wakeup livelock, a failed activation deactivates the cache rather
than leaving it active-but-invalid.

Dirty marking for these pages, which the CPU writes via the vmcs02
physical addresses and thus bypasses EPT-based dirty tracking, moves
from the unmap path to gfn-based marking from the vmcs12 fields on
every L2 exit (__vmx_handle_exit() already does this on every exit
from L2, so unmap-time marking was redundant); the one software
writer, __kvm_apic_update_irr() from
vmx_complete_nested_posted_interrupt(), marks at the write site.

¹ https://lore.kernel.org/all/ZBEEQtmtNPaEqU1i@xxxxxxxxxx

Signed-off-by: Fred Griffoul <fgriffo@xxxxxxxxxxxx>
Co-developed-by: David Woodhouse <dwmw@xxxxxxxxxxxx>
Failure to resolve a page distinguishes three cases. At entry time
(VMLAUNCH/VMRESUME, or resume from KVM_SET_NESTED_STATE, tracked in
nested.resume_pending) an unbackable page is a misconfiguration:
report KVM_EXIT_INTERNAL_ERROR as ever (vmx_apic_access_test asserts
this). On lazy revalidation — a pending KVM_REQ_GET_NESTED_STATE_PAGES
consumed while L2 runs — a slot mid-update (-EAGAIN from the
pfncache) re-posts the request and continues; the pending request
prevents VM entry, so the vCPU retries until the memslot update
completes. And a genuinely unbacked gfn on lazy revalidation means
userspace yanked the backing from under a running L2: clear
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES in vmcs02 instead of killing
the VM, so L2's accesses take ordinary EPT violations and are
handled as MMIO, exactly as a non-nested guest touching the same
unbacked gpa would be — and as upstream KVM (which never re-resolves)
effectively behaves. The control is recomputed from vmcs12 on the
next nested entry, and a later memslot update which re-backs the gfn
re-latches the page, so the degradation heals itself
(vmx_apic_update_test's move-memslot phase exercises this).

Signed-off-by: David Woodhouse <dwmw@xxxxxxxxxxxx>
Assisted-by: Claude:claude-mythos-5
---
 arch/x86/kvm/vmx/nested.c | 345 +++++++++++++++++++++++++++++++++-----
 arch/x86/kvm/vmx/vmx.c    |  11 +-
 arch/x86/kvm/vmx/vmx.h    |  14 +-
 3 files changed, 319 insertions(+), 51 deletions(-)

diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 5fe7e5d1f72d..105ff6cbfe58 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -352,14 +352,107 @@ static void nested_gpc_unlock(struct gfn_to_pfn_cache *gpc, int idx)
 	srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx);
 }
 
+/*
+ * Map a page of L1 memory, pin it for direct use by the guest (i.e. by
+ * the CPU while running the L2 guest, via a physical address in vmcs02),
+ * and return the host physical address to write into vmcs02. The pin
+ * guarantees that if the mapping is invalidated, this vCPU is forced out
+ * of guest mode and KVM_REQ_GET_NESTED_STATE_PAGES is posted so that the
+ * address is re-established (or the vCPU exits to userspace) before the
+ * next entry to guest mode.
+ */
+static int nested_gpc_hpa(struct gfn_to_pfn_cache *gpc, gpa_t gpa, hpa_t *hpa)
+{
+	int idx;
+
+	do {
+		idx = nested_gpc_lock(gpc, gpa);
+		if (idx < 0)
+			return idx;
+
+		/*
+		 * The pin can fail only if an invalidation cleared the
+		 * valid bit after nested_gpc_lock() checked it; go back
+		 * around to refresh (which will not complete until the
+		 * invalidation is over) and try again.
+		 */
+		if (kvm_gpc_pin_for_guest(gpc))
+			break;
+
+		nested_gpc_unlock(gpc, idx);
+	} while (1);
+
+	*hpa = pfn_to_hpa(gpc->pfn);
+	nested_gpc_unlock(gpc, idx);
+	return 0;
+}
+
+/*
+ * Try to lock an already-configured cache for host-side (khva) access,
+ * without sleeping. Returns -ENOENT if the cache is not active (e.g.
+ * posted interrupts not configured for this L2), -EWOULDBLOCK if it is
+ * active but currently invalidated (only a sleeping refresh would make
+ * it usable). Does not pin; the mapping is stable only until
+ * nested_gpc_unlock().
+ */
+static int nested_gpc_try_lock_if_active(struct gfn_to_pfn_cache *gpc)
+{
+	int idx;
+
+	/*
+	 * Check for an inactive (or never-initialized: gpc->kvm is NULL
+	 * before VMXON) cache before dereferencing gpc->kvm. Deactivation
+	 * only happens on this vCPU (under vcpu->mutex), so the check
+	 * cannot race with the lock which follows it.
+	 */
+	if (!gpc->active)
+		return -ENOENT;
+
+	idx = srcu_read_lock_atomic(&gpc->kvm->gpc_srcu);
+	if (!gpc->active) {
+		srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx);
+		return -ENOENT;
+	}
+
+	if (!kvm_gpc_check(gpc, PAGE_SIZE)) {
+		srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx);
+		return -EWOULDBLOCK;
+	}
+
+	return idx;
+}
+
+/*
+ * As above, but refresh the cache if it has been invalidated. May sleep;
+ * callers must be in a sleepable context (in particular, NOT under
+ * set_current_state() in a block loop, and not in a VM-exit fastpath).
+ */
+static int nested_gpc_lock_if_active(struct gfn_to_pfn_cache *gpc)
+{
+	int idx, err;
+
+	while ((idx = nested_gpc_try_lock_if_active(gpc)) == -EWOULDBLOCK) {
+		err = kvm_gpc_refresh(gpc, PAGE_SIZE);
+		if (err)
+			return err;
+	}
+
+	return idx;
+}
+
+static struct pi_desc *nested_pi_desc(struct vcpu_vmx *vmx)
+{
+	return (struct pi_desc *)((u8 *)vmx->nested.pi_desc_cache.khva +
+				  vmx->nested.pi_desc_offset);
+}
+
 static void nested_put_vmcs12_pages(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 
-	kvm_vcpu_unmap(vcpu, &vmx->nested.apic_access_page_map);
-	kvm_vcpu_unmap(vcpu, &vmx->nested.virtual_apic_map);
-	kvm_vcpu_unmap(vcpu, &vmx->nested.pi_desc_map);
-	vmx->nested.pi_desc = NULL;
+	kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache);
+	kvm_gpc_unpin_for_guest(&vmx->nested.virtual_apic_cache);
+	kvm_gpc_unpin_for_guest(&vmx->nested.pi_desc_cache);
 }
 
 /*
@@ -378,6 +471,7 @@ static void free_nested(struct kvm_vcpu *vcpu)
 		return;
 
 	kvm_clear_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+	vmx->nested.resume_pending = false;
 
 	vmx->nested.vmxon = false;
 	vmx->nested.smm.vmxon = false;
@@ -402,10 +496,13 @@ static void free_nested(struct kvm_vcpu *vcpu)
 	kfree(vmx->nested.cached_shadow_vmcs12);
 	vmx->nested.cached_shadow_vmcs12 = NULL;
 
-	kvm_gpc_deactivate(&vmx->nested.msr_bitmap_cache);
-
 	nested_put_vmcs12_pages(vcpu);
 
+	kvm_gpc_deactivate(&vmx->nested.pi_desc_cache);
+	kvm_gpc_deactivate(&vmx->nested.virtual_apic_cache);
+	kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache);
+	kvm_gpc_deactivate(&vmx->nested.msr_bitmap_cache);
+
 	kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL);
 
 	nested_release_evmcs(vcpu);
@@ -3475,11 +3572,11 @@ static bool nested_get_evmcs_page(struct kvm_vcpu *vcpu)
 }
 #endif
 
-static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
+static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu, bool lazy)
 {
 	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
-	struct kvm_host_map *map;
+	hpa_t hpa;
 
 	if (!vcpu->arch.pdptrs_from_userspace &&
 	    !nested_cpu_has_ept(vmcs12) && is_pae_paging(vcpu)) {
@@ -3494,26 +3591,74 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
 
 
 	if (nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) {
-		map = &vmx->nested.apic_access_page_map;
-
-		if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->apic_access_addr), map)) {
-			vmcs_write64(APIC_ACCESS_ADDR, pfn_to_hpa(map->pfn));
-		} else {
+		int err = nested_gpc_hpa(&vmx->nested.apic_access_page_cache,
+					 vmcs12->apic_access_addr, &hpa);
+		if (!err) {
+			vmcs_write64(APIC_ACCESS_ADDR, hpa);
+		} else if (err == -EAGAIN) {
+			kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache);
+			kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache);
+			goto retry;
+		} else if (!lazy) {
+			/*
+			 * Entry-time resolution (VMLAUNCH/VMRESUME, or resume
+			 * from KVM_SET_NESTED_STATE): an unbackable
+			 * APIC-access page is a misconfiguration to report,
+			 * not a transient to ride out.
+			 */
 			pr_debug_ratelimited("%s: no backing for APIC-access address in vmcs12\n",
 					     __func__);
 			vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
 			vcpu->run->internal.suberror =
 				KVM_INTERNAL_ERROR_EMULATION;
 			vcpu->run->internal.ndata = 0;
+			kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache);
+			kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache);
 			return false;
+		} else {
+			/*
+			 * Lazy revalidation (a memslot update or invalidation
+			 * while L2 runs) found the gfn unbacked: userspace
+			 * yanked the backing from under a running L2. Turn
+			 * off the APIC-access match for vmcs02 instead of
+			 * killing the VM; L2's accesses to the page then take
+			 * ordinary EPT violations on an unbacked gfn and are
+			 * handled (or reported) as MMIO, exactly as a
+			 * non-nested guest touching the same gpa would be.
+			 * prepare_vmcs02_early() recomputes the control from
+			 * vmcs12 on the next nested entry, and a later
+			 * memslot update which re-backs the gfn re-posts the
+			 * request and re-latches the page, so the
+			 * degradation heals itself.
+			 */
+			kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache);
+			kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache);
+			secondary_exec_controls_clearbit(vmx,
+				SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES);
 		}
 	}
 
 	if (nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW)) {
-		map = &vmx->nested.virtual_apic_map;
+		int vapic_err = nested_gpc_hpa(&vmx->nested.virtual_apic_cache,
+					       vmcs12->virtual_apic_page_addr, &hpa);
 
-		if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->virtual_apic_page_addr), map)) {
-			vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, pfn_to_hpa(map->pfn));
+		/*
+		 * On failure, deactivate the cache: an active-but-invalid
+		 * cache would read as "might have an event" to the
+		 * non-sleeping check in vmx_has_nested_events(), causing
+		 * spurious wakeups for as long as the page remains
+		 * unbackable.
+		 */
+		if (vapic_err) {
+			kvm_gpc_unpin_for_guest(&vmx->nested.virtual_apic_cache);
+			kvm_gpc_deactivate(&vmx->nested.virtual_apic_cache);
+		}
+
+		if (vapic_err == -EAGAIN)
+			goto retry;
+
+		if (!vapic_err) {
+			vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, hpa);
 		} else if (nested_cpu_has(vmcs12, CPU_BASED_CR8_LOAD_EXITING) &&
 		           nested_cpu_has(vmcs12, CPU_BASED_CR8_STORE_EXITING) &&
 			   !nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) {
@@ -3536,14 +3681,13 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
 	}
 
 	if (nested_cpu_has_posted_intr(vmcs12)) {
-		map = &vmx->nested.pi_desc_map;
-
-		if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->posted_intr_desc_addr), map)) {
-			vmx->nested.pi_desc =
-				(struct pi_desc *)(((void *)map->hva) +
-				offset_in_page(vmcs12->posted_intr_desc_addr));
+		int pi_err = nested_gpc_hpa(&vmx->nested.pi_desc_cache,
+					    vmcs12->posted_intr_desc_addr & PAGE_MASK, &hpa);
+		if (!pi_err) {
+			vmx->nested.pi_desc_offset =
+				offset_in_page(vmcs12->posted_intr_desc_addr);
 			vmcs_write64(POSTED_INTR_DESC_ADDR,
-				     pfn_to_hpa(map->pfn) + offset_in_page(vmcs12->posted_intr_desc_addr));
+				     hpa + offset_in_page(vmcs12->posted_intr_desc_addr));
 		} else {
 			/*
 			 * Defer the KVM_INTERNAL_EXIT until KVM tries to
@@ -3551,7 +3695,10 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
 			 * descriptor. (Note that KVM may do this when it
 			 * should not, per the architectural specification.)
 			 */
-			vmx->nested.pi_desc = NULL;
+			kvm_gpc_unpin_for_guest(&vmx->nested.pi_desc_cache);
+			kvm_gpc_deactivate(&vmx->nested.pi_desc_cache);
+			if (pi_err == -EAGAIN)
+				goto retry;
 			pin_controls_clearbit(vmx, PIN_BASED_POSTED_INTR);
 		}
 	}
@@ -3561,9 +3708,27 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu)
 		exec_controls_clearbit(vmx, CPU_BASED_USE_MSR_BITMAPS);
 
 	return true;
+
+retry:
+	/*
+	 * The gfn's memslot is being moved or deleted: present but
+	 * flagged invalid, so the lookup failure is transient and the
+	 * gfn must not (yet) be treated as unbacked. Re-post the
+	 * request and report success: the pending request prevents
+	 * any VM entry (vcpu_enter_guest() bails and reprocesses
+	 * requests), so L2 cannot run with the partially-updated
+	 * vmcs02, and the vCPU retries here until the memslot update
+	 * completes or reverts. This mirrors RET_PF_RETRY for
+	 * ordinary guest faults on an invalid slot; as there, a
+	 * userspace which never completes the update leaves the vCPU
+	 * retrying indefinitely, with signals and request processing
+	 * still serviced.
+	 */
+	kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+	return true;
 }
 
-static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
+static bool __vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
 {
 #ifdef CONFIG_KVM_HYPERV
 	/*
@@ -3584,9 +3749,39 @@ static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
 	}
 #endif
 
-	if (is_guest_mode(vcpu) && !nested_get_vmcs12_pages(vcpu))
+	if (is_guest_mode(vcpu) &&
+	    !nested_get_vmcs12_pages(vcpu, !to_vmx(vcpu)->nested.resume_pending))
 		return false;
 
+	/*
+	 * Success consumes the resume classification (a -EAGAIN retry
+	 * "succeeds" too: the resume did all it could against a memslot
+	 * mid-update, and the re-posted request's consumption is a fresh
+	 * revalidation like any other).
+	 */
+	to_vmx(vcpu)->nested.resume_pending = false;
+	return true;
+}
+
+static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu)
+{
+	if (!__vmx_get_nested_state_pages(vcpu)) {
+		/*
+		 * The caller (vcpu_enter_guest) consumed
+		 * KVM_REQ_GET_NESTED_STATE_PAGES before calling in, and a
+		 * failure here may leave a previously latched — and, after
+		 * an invalidation, stale and unreferenced — physical
+		 * address in vmcs02. Re-post the request so that no path
+		 * can re-enter L2 without retrying this function: the
+		 * invariant is that a stale latched address always implies
+		 * a pending request. If the failure persists, the vCPU
+		 * bounces to userspace on every KVM_RUN rather than ever
+		 * entering L2, which is the invariant doing its job.
+		 */
+		kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
+		return false;
+	}
+
 	return true;
 }
 
@@ -3715,7 +3910,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
 	prepare_vmcs02_early(vmx, &vmx->vmcs01, vmcs12);
 
 	if (from_vmentry) {
-		if (unlikely(!nested_get_vmcs12_pages(vcpu))) {
+		if (unlikely(!nested_get_vmcs12_pages(vcpu, false))) {
 			vmx_switch_vmcs(vcpu, &vmx->vmcs01);
 			return NVMX_VMENTRY_KVM_INTERNAL_ERROR;
 		}
@@ -3753,6 +3948,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
 		 * to nested_get_vmcs12_pages before the next VM-entry.  The MSRs
 		 * have already been set at vmentry time and should not be reset.
 		 */
+		vmx->nested.resume_pending = true;
 		kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu);
 	}
 
@@ -3809,6 +4005,14 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
 vmentry_fail_vmexit:
 	vmx_switch_vmcs(vcpu, &vmx->vmcs01);
 
+	/*
+	 * Release any guest-mode pins taken by nested_get_vmcs12_pages()
+	 * during this failed entry: this path returns to L1 without going
+	 * through __nested_vmx_vmexit(), which is where they are normally
+	 * dropped. (Idempotent if a later vmexit drops them again.)
+	 */
+	nested_put_vmcs12_pages(vcpu);
+
 	if (!from_vmentry)
 		return NVMX_VMENTRY_VMEXIT;
 
@@ -4065,6 +4269,8 @@ static void vmcs12_save_pending_event(struct kvm_vcpu *vcpu,
 static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	int pi_idx, vapic_idx;
+	struct pi_desc *pi_desc;
 	int max_irr;
 	void *vapic_page;
 	u16 status;
@@ -4072,22 +4278,32 @@ static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
 	if (!vmx->nested.pi_pending)
 		return 0;
 
-	if (!vmx->nested.pi_desc)
+	pi_idx = nested_gpc_lock_if_active(&vmx->nested.pi_desc_cache);
+	if (pi_idx < 0)
 		goto mmio_needed;
 
+	pi_desc = nested_pi_desc(vmx);
 	vmx->nested.pi_pending = false;
 
-	if (!pi_test_and_clear_on(vmx->nested.pi_desc))
+	if (!pi_test_and_clear_on(pi_desc)) {
+		nested_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx);
 		return 0;
+	}
 
-	max_irr = pi_find_highest_vector(vmx->nested.pi_desc);
+	max_irr = pi_find_highest_vector(pi_desc);
 	if (max_irr > 0) {
-		vapic_page = vmx->nested.virtual_apic_map.hva;
-		if (!vapic_page)
+		vapic_idx = nested_gpc_lock_if_active(&vmx->nested.virtual_apic_cache);
+		if (vapic_idx < 0) {
+			nested_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx);
 			goto mmio_needed;
+		}
+		vapic_page = vmx->nested.virtual_apic_cache.khva;
+
+		__kvm_apic_update_irr(pi_desc->pir, vapic_page, &max_irr);
+
+		kvm_gpc_mark_dirty_in_slot(&vmx->nested.virtual_apic_cache);
+		nested_gpc_unlock(&vmx->nested.virtual_apic_cache, vapic_idx);
 
-		__kvm_apic_update_irr(vmx->nested.pi_desc->pir,
-			vapic_page, &max_irr);
 		status = vmcs_read16(GUEST_INTR_STATUS);
 		if ((u8)max_irr > ((u8)status & 0xff)) {
 			status &= ~0xff;
@@ -4096,8 +4312,8 @@ static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu)
 		}
 	}
 
-	kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.virtual_apic_map);
-	kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.pi_desc_map);
+	kvm_gpc_mark_dirty_in_slot(&vmx->nested.pi_desc_cache);
+	nested_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx);
 	return 0;
 
 mmio_needed:
@@ -4216,8 +4432,9 @@ static bool nested_vmx_preemption_timer_pending(struct kvm_vcpu *vcpu)
 static bool vmx_has_nested_events(struct kvm_vcpu *vcpu, bool for_injection)
 {
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
-	void *vapic = vmx->nested.virtual_apic_map.hva;
-	int max_irr, vppr;
+	int max_irr, vppr, idx;
+	struct pi_desc *pi_desc;
+	void *vapic;
 
 	if (nested_vmx_preemption_timer_pending(vcpu) ||
 	    vmx->nested.mtf_pending)
@@ -4236,20 +4453,48 @@ static bool vmx_has_nested_events(struct kvm_vcpu *vcpu, bool for_injection)
 	    __vmx_interrupt_blocked(vcpu))
 		return false;
 
-	if (!vapic)
+	/*
+	 * This is called from non-sleeping contexts (notably the
+	 * kvm_vcpu_block() loop, under set_current_state()), so it must
+	 * not refresh an invalidated cache. Claim a pending event
+	 * instead: the bounce through vcpu_run services the pending
+	 * KVM_REQ_GET_NESTED_STATE_PAGES (posted by whatever invalidated
+	 * the cache) in a sleepable context, re-establishing the cache,
+	 * and re-evaluates events for real. A spurious wakeup is safe;
+	 * sleeping here is not, and returning false could miss a wakeup.
+	 */
+	idx = nested_gpc_try_lock_if_active(&vmx->nested.virtual_apic_cache);
+	if (idx == -EWOULDBLOCK)
+		return true;
+	if (idx < 0)
 		return false;
 
+	vapic = vmx->nested.virtual_apic_cache.khva;
 	vppr = *((u32 *)(vapic + APIC_PROCPRI));
 
+	nested_gpc_unlock(&vmx->nested.virtual_apic_cache, idx);
+
 	max_irr = vmx_get_rvi();
 	if ((max_irr & 0xf0) > (vppr & 0xf0))
 		return true;
 
-	if (vmx->nested.pi_pending && vmx->nested.pi_desc &&
-	    pi_test_on(vmx->nested.pi_desc)) {
-		max_irr = pi_find_highest_vector(vmx->nested.pi_desc);
-		if (max_irr > 0 && (max_irr & 0xf0) > (vppr & 0xf0))
+	if (vmx->nested.pi_pending) {
+		idx = nested_gpc_try_lock_if_active(&vmx->nested.pi_desc_cache);
+		if (idx == -EWOULDBLOCK)
 			return true;
+		if (idx < 0)
+			return false;
+
+		pi_desc = nested_pi_desc(vmx);
+		if (pi_test_on(pi_desc)) {
+			max_irr = pi_find_highest_vector(pi_desc);
+			if (max_irr > 0 && (max_irr & 0xf0) > (vppr & 0xf0)) {
+				nested_gpc_unlock(&vmx->nested.pi_desc_cache, idx);
+				return true;
+			}
+		}
+
+		nested_gpc_unlock(&vmx->nested.pi_desc_cache, idx);
 	}
 
 	return false;
@@ -5482,6 +5727,20 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu)
 	/* KVM only ever reads the L1 MSR bitmap, so never mark it dirty. */
 	__kvm_gpc_init(&vmx->nested.msr_bitmap_cache, vcpu->kvm, true, NULL, 0);
 
+	/*
+	 * The pfns of these three caches are handed to the CPU in vmcs02
+	 * fields, for direct use while running the L2 guest. Pinning them
+	 * for guest use makes an invalidation force this vCPU out of
+	 * guest mode and post KVM_REQ_GET_NESTED_STATE_PAGES, whose
+	 * handler re-establishes them before the next entry.
+	 */
+	kvm_gpc_init_for_vcpu(&vmx->nested.apic_access_page_cache, vcpu,
+			      KVM_REQ_GET_NESTED_STATE_PAGES);
+	kvm_gpc_init_for_vcpu(&vmx->nested.virtual_apic_cache, vcpu,
+			      KVM_REQ_GET_NESTED_STATE_PAGES);
+	kvm_gpc_init_for_vcpu(&vmx->nested.pi_desc_cache, vcpu,
+			      KVM_REQ_GET_NESTED_STATE_PAGES);
+
 	/*
 	 * Clear last_vpid to ensure that the VPID is flushed on the first
 	 * nested VM-Enter. Otherwise, stale TLB entries from a previous life of
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 612ab07d4100..25b0d114f2bf 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -6467,11 +6467,14 @@ static void vmx_flush_pml_buffer(struct kvm_vcpu *vcpu)
 
 static void nested_vmx_mark_all_vmcs12_pages_dirty(struct kvm_vcpu *vcpu)
 {
-	struct vcpu_vmx *vmx = to_vmx(vcpu);
+	struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
 
-	kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.apic_access_page_map);
-	kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.virtual_apic_map);
-	kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.pi_desc_map);
+	if (nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES))
+		kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->apic_access_addr));
+	if (nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW))
+		kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->virtual_apic_page_addr));
+	if (nested_cpu_has_posted_intr(vmcs12))
+		kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->posted_intr_desc_addr));
 }
 
 static void vmx_dump_sel(char *name, uint32_t sel)
diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h
index d5aa5aa83c95..f88e0031be4e 100644
--- a/arch/x86/kvm/vmx/vmx.h
+++ b/arch/x86/kvm/vmx/vmx.h
@@ -73,6 +73,12 @@ struct pt_desc {
 struct nested_vmx {
 	/* Has the level1 guest done vmxon? */
 	bool vmxon;
+	/*
+	 * The pending KVM_REQ_GET_NESTED_STATE_PAGES was posted by
+	 * KVM_SET_NESTED_STATE: an unbackable page is then a userspace
+	 * error to report, not a transient state to ride out.
+	 */
+	bool resume_pending;
 	gpa_t vmxon_ptr;
 	bool pml_full;
 
@@ -148,11 +154,11 @@ struct nested_vmx {
 	 */
 	struct gfn_to_pfn_cache msr_bitmap_cache;
 
-	struct kvm_host_map apic_access_page_map;
-	struct kvm_host_map virtual_apic_map;
-	struct kvm_host_map pi_desc_map;
+	struct gfn_to_pfn_cache apic_access_page_cache;
+	struct gfn_to_pfn_cache virtual_apic_cache;
+	struct gfn_to_pfn_cache pi_desc_cache;
 
-	struct pi_desc *pi_desc;
+	u64 pi_desc_offset;
 	bool pi_pending;
 	u16 posted_intr_nv;
 
-- 
2.55.0