Re: [PATCH v2 2/3] userfaultfd: UFFDIO_REMAP uABI

[Suren Baghdasaryan]

On Mon, Oct 2, 2023 at 12:34 PM Lokesh Gidra <lokeshgidra@xxxxxxxxxx> wrote:
>
> On Mon, Oct 2, 2023 at 6:43 PM David Hildenbrand <david@xxxxxxxxxx> wrote:
> >
> > On 02.10.23 17:55, Lokesh Gidra wrote:
> > > On Mon, Oct 2, 2023 at 4:46 PM Lokesh Gidra <lokeshgidra@xxxxxxxxxx> wrote:
> > >>
> > >> On Mon, Oct 2, 2023 at 4:21 PM Peter Xu <peterx@xxxxxxxxxx> wrote:
> > >>>
> > >>> On Mon, Oct 02, 2023 at 10:00:03AM +0200, David Hildenbrand wrote:
> > >>>> In case we cannot simply remap the page, the fallback sequence (from the
> > >>>> cover letter) would be triggered.
> > >>>>
> > >>>> 1) UFFDIO_COPY
> > >>>> 2) MADV_DONTNEED
> > >>>>
> > >>>> So we would just handle the operation internally without a fallback.
> > >>>
> > >>> Note that I think there will be a slight difference on whole remap
> > >>> atomicity, on what happens if the page is modified after UFFDIO_COPY but
> > >>> before DONTNEED.
> > >>>
> > >>> UFFDIO_REMAP guarantees full atomicity when moving the page, IOW, threads
> > >>> can be updating the pages when ioctl(UFFDIO_REMAP), data won't get lost
> > >>> during movement, and it will generate a missing event after moved, with
> > >>> latest data showing up on dest.
> > >>>
> > >>> I'm not sure that means such a fallback is a problem, Suren may know
> > >>> better with the use case.
> > >>
> > >> Although there is no problem in using fallback with our use case but
> > >> as a user of userfaultfd, I'd suggest leaving it to the developer.
> > >> Failing with appropriate errno makes more sense. If handled in the
> > >> kernel, then the user may assume at the end of the operation that the
> > >> src vma is completely unmapped. And if not correctness issues, it
> > >> could lead to memory leaks.
> > >
> > > I meant that in addition to the possibility of correctness issues due
> > > to lack of atomicity, it could also lead to memory leaks, as the user
> > > may assume that src vma is empty post-operation. IMHO, it's better to
> > > fail with errno so that the user would fix the code with necessary
> > > changes (like using DONTFORK, if forking).
> >
> > Leaving the atomicity discussion out because I think this can just be
> > handled (e.g., the src_vma would always be empty post-operation):
> >
> > It might not necessarily be a good idea to only expose micro-operations
> > to user space. If the user-space fallback will almost always be
> > "UFFDIO_COPY+MADV_DONTNEED", then clearly the logical operation
> > performed is moving data, ideally with zero-copy.
> >
> IMHO, such a fallback will be useful only if it's possible that only
> some pages in the src vma fail due to this. But even then it would be
> really useful to have a flag maybe like UFFDIO_REMAP_FALLBACK_COPY to
> control if the user wants the fallback or not. OTOH, if this is
> something that can be detected for the entire src vma, then failing
> with errno is more appropriate.
>
> Given that the patch is already quite complicated, I humbly suggest
> leaving the fallback for now as a TODO.

Ok, I think it makes sense to implement the strict remap logic but in
a way that we can easily add copy fallback if that's needed in the
future. So, I'll change UFFDIO_REMAP to UFFDIO_MOVE and will return
some unique error, like EBUSY when the page is not PAE. If we need to
add a copy fallback in the future, we will add a
UFFDIO_MOVE_MODE_ALLOW_COPY flag and will implement the copy
mechanism. Does that sound good?

>
> > [as said as reply to Peter, one could still have magic flags for users
> > that really want to detect when zero-copy is impossible]
> >
> > With a logical MOVE API users like compaction [as given in the cover
> > letter], not every such user has to eventually implement fallback paths.
> >
> > But just my 2 cents, the UFFDIO_REMAP users probably can share what the
> > exact use cases are and if fallbacks are required at all or if no-KSM +
> > DONTFORK just does the trick.
> >
> > --
> > Cheers,
> >
> > David / dhildenb
> >