Re: [PATCH 0/7] mm/userfaultfd/poll: Scale userfaultfd wakeups

[Peter Xu]

On Thu, Sep 07, 2023 at 12:18:29PM -0700, Axel Rasmussen wrote:
> On Tue, Sep 5, 2023 at 2:42 PM Peter Xu <peterx@xxxxxxxxxx> wrote:
> >
> > Userfaultfd is the type of file that doesn't need wake-all semantics: if
> > there is a message enqueued (for either a fault address, or an event), we
> > only need to wake up one service thread to handle it.  Waking up more
> > normally means a waste of cpu cycles.  Besides that, and more importantly,
> > that just doesn't scale.
> 
> Hi Peter,

Hi, Axel,

Sorry to respond late.

> 
> I took a quick look over the series and didn't see anything
> objectionable. I was planning to actually test the series out and then
> send out R-b's, but it will take some additional time (next week).

Thanks.  The 2nd patch definitely needs some fixup on some functions
(either I overlooked without enough CONFIG_* chosen; I am surprised I have
vhost even compiled out when testing..), hope that won't bring you too much
trouble.  I'll send a fixup soon on top of patch 2.

> 
> In the meantime, I was curious about the use case. A design I've seen
> for VM live migration is to have 1 thread reading events off the uffd,
> and then have many threads actually resolving the fault events that
> come in (e.g. fetching pages over the network, issuing UFFDIO_COPY or
> UFFDIO_CONTINUE, or whatever). In that design, since we only have a
> single reader anyway, I think this series doesn't help.

Yes.  If the test to carry out only uses 1 thread, it shouldn't bring much
difference.

> 
> But, I'm curious if you have data indicating that > 1 reader is more
> performant overall? I suspect it might be the case that, with "enough"
> vCPUs, it makes sense to do so, but I don't have benchmark data to
> tell me what that tipping point is yet.
> 
> OTOH, if one reader is plenty in ~all cases, optimizing this path is
> less important.

For myself I don't yet have an application that can leverage this much
indeed, because QEMU so far only uses 1 reader thread.

IIRC Anish was exactly proposing some kvm specific solutions to make single
uffd scale, and this might be suitable for any use case like that where we
may want to use single uffd and try to make it scale with threads.  Using 1
reader + N worker is also a solution, but when using N readers (which also
do the work) the app will hit this problem.

I am also aware that some apps use more than 1 reader threads (umap), but I
don't really know more than that.

The problem is I think we shouldn't have that overhead easily just because
an app invokes >1 readers, meanwhile it also doesn't make much sense to
wake up all readers for a single event for userfaults.  So it should always
be something good to have.

Thanks,

-- 
Peter Xu