Re: [RFC] KVM: x86/mmu: Prefetch forward run of pages on TDP page faults

[Marangoni, Marco]

On Tue, Aug 25, 2026, James Houghton <jthoughton@xxxxxxxxxx> wrote:
>
> On Tue, Aug 25, 2026 at 10:38 AM Marangoni, Marco <mamarang@xxxxxxxxxx> wrote:
> >
> > Thanks for the replies!
> >
> > On Tue, Aug 25, 2026, Sean Christopherson <seanjc@xxxxxxxxxx> wrote:
> > >
> > > On Tue, Aug 25, 2026, Marco Marangoni wrote:
> > > > It's worth mentioning that I also evaluated using the existing
> > > > KVM_PRE_FAULT_MEMORY ioctl, but this doesn't work well for our use-case, as
> > > > it requires the vCPU to be paused.
> > >
> > > What about if/when KVM Userfault[*] comes along?  I.e. pre-fault memory when the
> > > vCPU exits to userspace.
> >
> > KVM Userfault + KVM_PRE_FAULT_MEMORY is a valid suggestion, however if possible we'd like to have _both_ async page faults and prefetching.
> > I haven't tested async PF together with prefetching, but tested separately, both improvements yield great results, so it would be a shame to have to choose.
> 
> What if you used KVM_PRE_FAULT_MEMORY without KVM Userfault?
> 
> If you want to avoid pausing a vCPU, what if you made another vCPU
> (KVM_CREATE_VCPU) and used that solely for prefaulting guest memory? I
> haven't really looked into that before... I'm guessing there's
> something fundamentally wrong with this approach.
> 
> If this doesn't work (and a VM-scoped KVM_PRE_FAULT_MEMORY doesn't
> make sense either), then perhaps the TDP MMU prefetching logic makes
> sense.

~~Unfortunately~~ Fortunately I'll be on Holidays for the next few weeks; I'll try the permanently paused vCPU approach as soon as I can.
In principle it could solve the problem without requiring any kernel changes.


On Tue, Aug 25, 2026, James Houghton <jthoughton@xxxxxxxxxx> wrote:
>
> > On Tue, Aug 25, 2026, James Houghton <jthoughton@xxxxxxxxxx> wrote:
> > > I think part of the problem in this case is that UFFDIO_COPY will
> > > install 4K pages (IIRC), I think a more natural way to fix this
> > > problem is to:
> > >
> > > 1. MADV_COLLAPSE after doing UFFDIO_COPY.
> > > 2. Make UFFDIO_COPY install PMDs when it is able to do so.
> > >
> > > These don't solve the exact same problem, but really userfaultfd
> > > should already try to install PMDs when it can (#2). If we have #2, #1
> > > is mostly a no-op.
> > >
> > > What do you think?
> >
> > Directly installing PMDs after an UFFD_COPY is something I already investigated. I didn't mention it originally, since it touches exclusively the MM module.
> > For some context, with that approach, in the same benchmarks, fault latency on nested is reduced by 92%, and by 44% on metal, which is significantly better than my > proposal (which "only" improves by 75% and 22% respectively).
> 
> Did you or one of your colleagues ever post it on list? I'm curious to
> see it. :)

I wanted to tackle the use-case of arbitrary UFFD chunks first, so no, not yet at least.
If you're curious, the proof-of-concept is on https://github.com/marco-marangoni/firecracker/blob/thp-uffd/resources/patches/thp_uffd.patch; it lacks mTHP support and only handles UFFD_COPY.


On Tue, Aug 25, 2026, James Houghton <jthoughton@xxxxxxxxxx> wrote:
>
> > However, that approach only works when the copy is done in multiples of 2MiB, and for some Firecracker use-cases, that's a no-go (I can elaborate further if necessary, but the main problem is an explosion in incremental snapshots size when managing memory in big chunks).
> > I might pursue this proposal in a separate patch, however I'd love to work out a solution that can be applied when userfaultfd works with smaller chunk sizes.
> 
> I see. So we really are dealing with 4K mappings.
> 
> In which case, prefetching at 2M does seem kind of arbitrary, which
> makes me even more in favor of this being mostly userspace-driven.

Yes, 2MiB is arbitrary. 
One of the possible implementation for this prefetching patch is to skip the GUP and create the EPT PT directly from the page table, in which case 2MiB is natural.


Thanks again for the interesting replies, I'll try the KVM_PRE_FAULT_MEMORY with a paused vCPU when I'm back from PTO.

Marco