Re: [PATCH 1/2] mm: introduce put_user_page*(), placeholder versions

[Dan Williams]

On Mon, Dec 17, 2018 at 12:57 AM Jan Kara <jack@xxxxxxx> wrote:
>
> On Fri 14-12-18 11:38:59, Dan Williams wrote:
> > On Thu, Dec 13, 2018 at 10:11 PM John Hubbard <jhubbard@xxxxxxxxxx> wrote:
> > >
> > > On 12/13/18 9:21 PM, Dan Williams wrote:
> > > > On Thu, Dec 13, 2018 at 7:53 PM John Hubbard <jhubbard@xxxxxxxxxx> wrote:
> > > >>
> > > >> On 12/12/18 4:51 PM, Dave Chinner wrote:
> > > >>> On Wed, Dec 12, 2018 at 04:59:31PM -0500, Jerome Glisse wrote:
> > > >>>> On Thu, Dec 13, 2018 at 08:46:41AM +1100, Dave Chinner wrote:
> > > >>>>> On Wed, Dec 12, 2018 at 10:03:20AM -0500, Jerome Glisse wrote:
> > > >>>>>> On Mon, Dec 10, 2018 at 11:28:46AM +0100, Jan Kara wrote:
> > > >>>>>>> On Fri 07-12-18 21:24:46, Jerome Glisse wrote:
> > > >>>>>>> So this approach doesn't look like a win to me over using counter in struct
> > > >>>>>>> page and I'd rather try looking into squeezing HMM public page usage of
> > > >>>>>>> struct page so that we can fit that gup counter there as well. I know that
> > > >>>>>>> it may be easier said than done...
> > > >>>>>>
> > > >>
> > > >> Agreed. After all the discussion this week, I'm thinking that the original idea
> > > >> of a per-struct-page counter is better. Fortunately, we can do the moral equivalent
> > > >> of that, unless I'm overlooking something: Jerome had another proposal that he
> > > >> described, off-list, for doing that counting, and his idea avoids the problem of
> > > >> finding space in struct page. (And in fact, when I responded yesterday, I initially
> > > >> thought that's where he was going with this.)
> > > >>
> > > >> So how about this hybrid solution:
> > > >>
> > > >> 1. Stay with the basic RFC approach of using a per-page counter, but actually
> > > >> store the counter(s) in the mappings instead of the struct page. We can use
> > > >> !PageAnon and page_mapping to look up all the mappings, stash the dma_pinned_count
> > > >> there. So the total pinned count is scattered across mappings. Probably still need
> > > >> a PageDmaPinned bit.
> > > >
> > > > How do you safely look at page->mapping from the get_user_pages_fast()
> > > > path? You'll be racing invalidation disconnecting the page from the
> > > > mapping.
> > > >
> > >
> > > I don't have an answer for that, so maybe the page->mapping idea is dead already.
> > >
> > > So in that case, there is still one more way to do all of this, which is to
> > > combine ZONE_DEVICE, HMM, and gup/dma information in a per-page struct, and get
> > > there via basically page->private, more or less like this:
> >
> > If we're going to allocate something new out-of-line then maybe we
> > should go even further to allow for a page "proxy" object to front a
> > real struct page. This idea arose from Dave Hansen as I explained to
> > him the dax-reflink problem, and dovetails with Dave Chinner's
> > suggestion earlier in this thread for dax-reflink.
> >
> > Have get_user_pages() allocate a proxy object that gets passed around
> > to drivers. Something like a struct page pointer with bit 0 set. This
> > would add a conditional branch and pointer chase to many page
> > operations, like page_to_pfn(), I thought something like it would be
> > unacceptable a few years ago, but then HMM went and added similar
> > overhead to put_page() and nobody balked.
> >
> > This has the additional benefit of catching cases that might be doing
> > a get_page() on a get_user_pages() result and should instead switch to
> > a "ref_user_page()" (opposite of put_user_page()) as the API to take
> > additional references on a get_user_pages() result.
> >
> > page->index and page->mapping could be overridden by similar
> > attributes in the proxy, and allow an N:1 relationship of proxy
> > instances to actual pages. Filesystems could generate dynamic proxies
> > as well.
> >
> > The auxiliary information (dev_pagemap, hmm_data, etc...) moves to the
> > proxy and stops polluting the base struct page which remains the
> > canonical location for dirty-tracking and dma operations.
> >
> > The difficulties are reconciling the source of the proxies as both
> > get_user_pages() and filesystem may want to be the source of the
> > allocation. In the get_user_pages_fast() path we may not be able to
> > ask the filesystem for the proxy, at least not without destroying the
> > performance expectations of get_user_pages_fast().
>
> What you describe here sounds almost like page_ext mechanism we already
> have? Or do you really aim at per-pin allocated structure?

Per-pin or dynamically allocated by the filesystem. The existing
page_ext seems to suffer from the expectation that a page_ext exists
for all pfns. The 'struct page' per pfn requirement is already painful
as memory capacities grow into the terabytes, page_ext seems to just
make that worse.