Re: [PATCH] KVM: guest_memfd: Disable VMA merging with VM_DONTEXPAND

[David Hildenbrand (arm)]

On 2/4/26 22:37, Sean Christopherson wrote:
> On Wed, Feb 04, 2026, Ackerley Tng wrote:
> > Ackerley Tng <ackerleytng@xxxxxxxxxx> writes:
> >
> > > #syz test: git://git.kernel.org/pub/scm/virt/kvm/kvm.git next
> > >
> > > guest_memfd VMAs don't need to be merged,
>
> Why not?  There are benefits to merging VMAs that have nothing to do with folios.
> E.g. map 1GiB of guest_memfd with 512*512 4KiB VMAs, and then it becomes quite
> desirable to merge all of those VMAs into one.
>
> Creating _hugepages_ doesn't add value, but that's not the same things as merging
> VMAs.
>
> > > especially now, since guest_memfd only supports PAGE_SIZE folios.
> > >
> > > Set VM_DONTEXPAND on guest_memfd VMAs.
> >
> > Local tests and syzbot agree that this fixes the issue identified. :)
> >
> > I would like to look into madvise(MADV_COLLAPSE) and uprobes triggering
> > mapping/folio collapsing before submitting a full patch series.
> >
> > David, Michael, Vishal, what do you think of the choice of setting
> > VM_DONTEXPAND to disable khugepaged?
>
> I'm not one of the above, but for me it feels very much like treating a symptom
> and not fixing the underlying cause.

And you are spot-on :)

> It seems like what KVM should do is not block one path that triggers hugepage
> processing, but instead flat out disallow creating hugepages.  Unfortunately,
> AFAICT, there's no existing way to prevent madvise() from clearing VM_NOHUGEPAGE,
> so we can't simply force that flag.
>
> I'd prefer not to special case guest_memfd, a la devdax, but I also want to address
> this head-on, not by removing a tangentially related trigger.

VM_NOHUGEPAGE also smells like the wrong thing. This is a file limitation.

!thp_vma_allowable_order() must take care of that somehow down in 
__thp_vma_allowable_orders(), by checking the file).

Likely the file_thp_enabled() check is the culprit with 
CONFIG_READ_ONLY_THP_FOR_FS?

Maybe we need a flag to say "even not CONFIG_READ_ONLY_THP_FOR_FS".

I wonder how we handle that for secretmem. Too late for me, going to bed :)

--
Cheers,

David