Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[James Houghton]

On Tue, Aug 25, 2026 at 10:38 AM Marangoni, Marco <mamarang@xxxxxxxxxx> wrote:
>
> Thanks for the replies!
>
> On Tue, Aug 25, 2026, Sean Christopherson <seanjc@xxxxxxxxxx> wrote:
> >
> > On Tue, Aug 25, 2026, Marco Marangoni wrote:
> > > It's worth mentioning that I also evaluated using the existing
> > > KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> > > it requires the vCPU to be paused.
> >
> > What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
> > vCPU exits to userspace.
>
> KVM Userfault + KVM_PRE_FAULT_MEMORY is a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching.
> I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.

What if you used KVM_PRE_FAULT_MEMORY without KVM Userfault?

If you want to avoid pausing a vCPU, what if you made another vCPU
(KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I
haven't really looked into that before... I'm guessing there's
something fundamentally wrong with this approach.

If this doesn't work (and a VM-scoped KVM_PRE_FAULT_MEMORY doesn't
make sense either), then perhaps the TDP MMU prefetching logic makes
sense.

> On Tue, Aug 25, 2026, James Houghton <jthoughton@xxxxxxxxxx> wrote:
> > I think part of the problem in this case is that UFFDIO_COPY will
> > install 4K pages (IIRC), I think a more natural way to fix this
> > problem is to:
> >
> > 1. MADV_COLLAPSE after doing UFFDIO_COPY.
> > 2. Make UFFDIO_COPY install PMDs when it is able to do so.
> >
> > These don't solve the exact same problem, but really userfaultfd
> > should already try to install PMDs when it can (#2). If we have #2, #1
> > is mostly a no-op.
> >
> > What do you think?
>
> Directly installing PMDs after an UFFD_COPY is something I already investigated. I didn't mention it originally, since it touches exclusively the MM module.
> For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my proposal (which "only" improves by 75% and 22% respectively).

Did you or one of your colleagues ever post it on list? I'm curious to
see it. :)

> However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks).
> I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.

I see. So we really are dealing with 4K mappings.

In which case, prefetching at 2M does seem kind of arbitrary, which
makes me even more in favor of this being mostly userspace-driven.