Re: [PATCH v3 03/26] mm: introduce AS_NO_DIRECT_MAP

[Yosry Ahmed]

> > > > > > > At that point, userspace is basically required to
> > > > > > > maintain mappings for all host-accessible guest memory, and if there are userspace
> > > > > > > mappings, then not using GUP doesn't make much sense.
> > > > > > >
> > > > > > > Note, I called out x86 because x86 has the most extensive emulator and shadow
> > > > > > > paging support, which is where the isolated, one-off accesses happen in spades.
> > > > > > > Other architectures might be able to squeak by without userspace mappings, at
> > > > > > > least for now.
> > > > > > >
> > > > > > > So, in all likelihood, KVM will want GUP.
> > > > > >
> > > > > > Yeah I am thinking that the check here to disallow GUP completely for
> > > > > > unmapped pages is aggressive. Maybe it works for now if KVM does not
> > > > > > currently have any use cases for accessing guest_memfd memory. But if it does
> > > > > > (or will very soon), we need to think more about it, otherwise
> > > > > > AS_NO_DIRECT_MAP is not really usable for guest_memfd. Since you said KVM
> > > > > > "will want" GUP, I assume it currently doesn't?
> > > > >
> > > > > Doesn't what?  Have GUP?  KVM heavily uses GUP, including for guest_memfd that
> > > > > can be mapped into userspace.
> > > >
> > > > Your wording made me think that KVM doesn't currently use GUP for
> > > > guest_memfd, but I was obviously wrong. So IIUC GUP needs to succeed for
> > > > guest_memfd pages with AS_NO_DIRECT_MAP.
> > >
> > > Yes, though as I said early, it doesn't *have* to be exactly GUP, just something
> > > GUP-like.  E.g. it could be a new API, if that's easier/cleaner.  What I don't
> > > think is a good idea though is handling this entirely in KVM/guest_memfd.
> >
> > Just to clarify, you mean that GUP (or GUP-like) should work in terms of
> > pinning the page and handing KVM/guest_memfd the pfn/address, but not
> > actually making the page accessible or establishing mappings, right?
>
> No, I'm saying that whatever API the kernel provides needs to ensure there's a
> valid kernel mapping (or provide one as a return value).
>
> > Looking at [2], seems like the consensus was that AS_NO_DIRECT_MAP
> > means folios are not in the direct map, and callers are responsible
> > for establishing the mappings (e.g. using the mermap).
>
> I'm fine with that direction, but in that case GUP _does_ need to be disallowed.
> I.e. _if_ we allow GUP, then GUP itself needs to somehow ensure the direct map
> is populated.  If GUP is not allowed, then IMO the core kernel needs to provide
> an API to get at "inaccessible" mappings.  Or I suppose GUP could take a flag
> that says "I pinky-swear not to try and access the memory via the direct map".

Okay in this case I think the simplest thing to do here is to disallow
GUP for unmapped pages like this patch does. Once we have a use case
(e.g. guest_memfd with unmapped memory using GUP), we can figure out
if we want a separate API or a GUP flag. I assume the GUP flag could
either be: "I know the page is unmapped, I will map it" or "create an
ephemeral mapping for the page".

>
> > [2]https://lore.kernel.org/all/aeemS2wm38Cm4qAf@xxxxxxxxxx/
> >
> > >
> > > > To actually access the memory, I assume the guest_memfd side will need to
> > > > handle this by either using ephemeral mappings (e.g. mermap), restoring and
> > > > zapping direct mappings, or using a userspace mapping. I suppose for the
> > > > purposes of AS_NO_DIRECT_MAP core support we just need GUP to succeed?
> > >
> > > And establish a (ephemeral?) kernel mapping, because general users of GUP will
> > > expect that they can access the physical memory through the direct map.  That's
> > > why I didn't want to handle any of this in KVM[*], the rules and handling need
> > > to be kernel-wide.
> >
> > See above, I am struggling to understand where you think establishing
> > mappings should lie.
>
> Heh, I'm not surprised you're struggling, because I don't really have an opinion
> on exactly who/what is responsible for establishing the mappings.  What I care
> about at this point is not having guest_memfd itself provide a GUP-like API: that
> needs to be a generic kernel API.

Yeah that makes sense. I think the options discussed were basically:
(a) Alternate API to GUP.
(b) GUP + mermap (or similar)
(c) Just GUP (with mermap handled within)

I suspect (b) or (c).

>
> > The current approach is that AS_NO_DIRECT_MAP just means folios are not
> > mapped, and users are responsible for establishing the mappings. I assume you
> > agree with this (since you suggested this :P), but you don't want KVM to do
> > this ad-hoc, but to have a library for it.
> >
> > This library should be the mermap. I imagine (for e.g.) kvm_vcpu_map()
> > using the mermap under the hood if it knows the mappings do not exist
> > and using the mermap virtual address instead of the direct map
> > address. This only works (with the current implementation) if we can
> > disable migration (or even better, preemption) while a mapping is
> > active, which I imagine would be tricky or just not possible.
> >
> > The alternative could be destroying and recreating the mappings when
> > the vCPU moves between CPUs, which is.. interesting :)
> >
> > I imagine we don't have to sort all of this out now. For the purposes
> > of AS_NO_DIRECT_MAP (and secretmem AFAICT), we just need to provide a
> > facility to allocate unmapped pages. None of this is user-facing at
> > this point.
> >
> > >
> > > [*] https://lore.kernel.org/all/aeemS2wm38Cm4qAf@xxxxxxxxxx