Re: [PATCH] mm: introduce MADV_CLR_HUGEPAGE

[Mike Rapoport]

On Mon, May 22, 2017 at 03:55:48PM +0200, Michal Hocko wrote:
> On Mon 22-05-17 16:36:00, Mike Rapoport wrote:
> > On Mon, May 22, 2017 at 02:42:43PM +0300, Kirill A. Shutemov wrote:
> > > On Mon, May 22, 2017 at 09:12:42AM +0300, Mike Rapoport wrote:
> > > > Currently applications can explicitly enable or disable THP for a memory
> > > > region using MADV_HUGEPAGE or MADV_NOHUGEPAGE. However, once either of
> > > > these advises is used, the region will always have
> > > > VM_HUGEPAGE/VM_NOHUGEPAGE flag set in vma->vm_flags.
> > > > The MADV_CLR_HUGEPAGE resets both these flags and allows managing THP in
> > > > the region according to system-wide settings.
> > > 
> > > Seems reasonable. But could you describe an use-case when it's useful in
> > > real world.
> > 
> > My use-case was combination of pre- and post-copy migration of containers
> > with CRIU.
> > In this case we populate a part of a memory region with data that was saved
> > during the pre-copy stage. Afterwards, the region is registered with
> > userfaultfd and we expect to get page faults for the parts of the region
> > that were not yet populated. However, khugepaged collapses the pages and
> > the page faults we would expect do not occur.
> 
> I am not sure I undestand the problem. Do I get it right that the
> khugepaged will effectivelly corrupt the memory by collapsing a range
> which is not yet fully populated? If yes shouldn't that be fixed in
> khugepaged rather than adding yet another madvise command? Also how do
> you prevent on races? (say you VM_NOHUGEPAGE, khugepaged would be in the
> middle of the operation and sees a collapsable vma and you get the same
> result)

Probably I didn't explained it too well.

The range is intentionally not populated. When we combine pre- and
post-copy for process migration, we create memory pre-dump without stopping
the process, then we freeze the process without dumping the pages it has
dirtied between pre-dump and freeze, and then, during restore, we populate
the dirtied pages using userfaultfd.

When CRIU restores a process in such scenario, it does something like:

* mmap() memory region
* fill in the pages that were collected during the pre-dump
* do some other stuff
* register memory region with userfaultfd
* populate the missing memory on demand

khugepaged collapses the pages in the partially populated regions before we
have a chance to register these regions with userfaultfd, which would
prevent the collapse.

We could have used MADV_NOHUGEPAGE right after the mmap() call, and then
there would be no race because there would be nothing for khugepaged to
collapse at that point. But the problem is that we have no way to reset
*HUGEPAGE flags after the memory restore is complete.

> -- 
> Michal Hocko
> SUSE Labs

--
Sincerely yours,
Mike.