Re: [LSF/MM TOPIC]: userfaultfd (was: [LSF/MM TOPIC] NUMA remote THP vs NUMA local non-THP under MADV_HUGEPAGE)

[Peter Xu]

On Wed, Jan 30, 2019 at 10:13:36AM +0200, Mike Rapoport wrote:
> Hi,
> 
> (changed the subject and added CRIU folks)
> 
> On Tue, Jan 29, 2019 at 06:40:58PM -0500, Andrea Arcangeli wrote:
> > Hello,
> > 
> > --
> > 
> > In addition to the above "NUMA remote THP vs NUMA local non-THP
> > tradeoff" topic, there are other developments in "userfaultfd" land that
> > are approaching merge readiness and that would be possible to provide a
> > short overview about:
> > 
> > - Peter Xu made significant progress in finalizing the userfaultfd-WP
> >   support over the last few months. That feature was planned from the
> >   start and it will allow userland to do some new things that weren't
> >   possible to achieve before. In addition to synchronously blocking
> >   write faults to be resolved by an userland manager, it has also the
> >   ability to obsolete the softdirty feature, because it can provide
> >   the same information, but with O(1) complexity (as opposed of the
> >   current softdirty O(N) complexity) similarly to what the Page
> >   Modification Logging (PML) does in hardware for EPT write accesses.
>  
> We (CRIU) have some concerns about obsoleting soft-dirty in favor of
> uffd-wp. If there are other soft-dirty users these concerns would be
> relevant to them as well.
> 
> With soft-dirty we collect the information about the changed memory every
> pre-dump iteration in the following manner:
> * freeze the tasks
> * find entries in /proc/pid/pagemap with SOFT_DIRTY set
> * unfreeze the tasks
> * dump the modified pages to disk/remote host
> 
> While we do need to traverse the /proc/pid/pagemap to identify dirty pages,
> in between the pre-dump iterations and during the actual memory dump the
> tasks are running freely.
> 
> If we are to switch to uffd-wp, every write by the snapshotted/migrated
> task will incur latency of uffd-wp processing by the monitor.
> 
> We'd need to see how this affects overall slowdown of the workload under
> migration before moving forward with obsoleting soft-dirty.
> 
> > - Blake Caldwell maintained the UFFDIO_REMAP support to atomically
> >   remove memory from a mapping with userfaultfd (which can't be done
> >   with a copy as in UFFDIO_COPY and it requires a slow TLB flush to be
> >   safe) as an alternative to host swapping (which of course also
> >   requires a TLB flush for similar reasons). Notably UFFDIO_REMAP was
> >   rightfully naked early on and quickly replaced by UFFDIO_COPY which
> >   is more optimal to add memory to a mapping is small chunks, but we
> >   can't remove memory with UFFDIO_COPY and UFFDIO_REMAP should be as
> >   efficient as it gets when it comes to removing memory from a
> >   mapping.
> 
> If we are to discuss userfaultfd, I'd like also to bring the subject of COW
> mappings.
> The pages populated with UFFDIO_COPY cannot be COW-shared between related
> processes which unnecessarily increases memory footprint of a migrated
> process tree.
> I've posted a patch [1] a (real) while ago, but nobody reacted and I've put
> this aside.
> Maybe it's time to discuss it again :)

Hi, Mike,

It's interesting to know such a work...

Since I really don't have much context on this, so sorry if I'm going
to ask a silly question... but I'd say when reading this I'm thinking
of KSM.  I think KSM does not suite in this case since when doing
UFFDIO_COPY_COW it'll contain hinting information while KSM was only
scanning over the pages between processes which seems to be O(N*N) if
assuming there're two processes.  However, would it make any sense to
provide a general interface to scan for same pages between any two
processes within specific range and merge them if found (rather than a
specific interface for userfaultfd only)?  Then it might even be used
by KSM admins (just as an example) when the admin knows exactly that
memory range (addr1, len) of process A should very probably has many
same contents as the memory range (addr2, len) of process B?

Thanks,

-- 
Peter Xu