Re: [PATCH v1] fs/fuse: Fix missing FOLL_PIN for direct-io

[Lei Huang]

Hi Bernd,

Thank you very much for your reply!

> Hmm, iov_iter_extract_pages does not exists for a long time and the code
> in fuse_get_user_pages didn't change much. So if you are right, there
> would be a long term data corruption for page migrations? And a back
> port to old kernels would not be obvious?

Right. The issue has been reproduced under various versions of kernels, 
ranging from 3.10.0 to 6.3.6 in my tests. It would be different to make 
a patch under older kernels like 3.10.0. One way I tested, one can query
the physical pages associated with read buffer after data is ready 
(right before writing the data into read buffer). This seems resolving 
the issue in my tests.


> What confuses me further is that
> commit 85dd2c8ff368 does not mention migration or corruption, although
> lists several other advantages for iov_iter_extract_pages. Other commits
> using iov_iter_extract_pages point to fork - i.e. would your data
> corruption be possibly related that?

As I mentioned above, the issue seems resolved if we query the physical 
pages as late as right before writing data into read buffer. I think the 
root cause is page migration.

Best Regards,
-lei

On 8/29/23 17:57, Bernd Schubert wrote:
> On 8/29/23 20:36, Lei Huang wrote:
> > Our user space filesystem relies on fuse to provide POSIX interface.
> > In our test, a known string is written into a file and the content
> > is read back later to verify correct data returned. We observed wrong
> > data returned in read buffer in rare cases although correct data are
> > stored in our filesystem.
> >
> > Fuse kernel module calls iov_iter_get_pages2() to get the physical
> > pages of the user-space read buffer passed in read(). The pages are
> > not pinned to avoid page migration. When page migration occurs, the
> > consequence are two-folds.
> >
> > 1) Applications do not receive correct data in read buffer.
> > 2) fuse kernel writes data into a wrong place.
> >
> > Using iov_iter_extract_pages() to pin pages fixes the issue in our
> > test.
>
> Hmm, iov_iter_extract_pages does not exists for a long time and the code 
> in fuse_get_user_pages didn't change much. So if you are right, there 
> would be a long term data corruption for page migrations? And a back 
> port to old kernels would not be obvious?
>
> What confuses me further is that
> commit 85dd2c8ff368 does not mention migration or corruption, although 
> lists several other advantages for iov_iter_extract_pages. Other commits 
> using iov_iter_extract_pages point to fork - i.e. would your data 
> corruption be possibly related that?
>
>
> Thanks,
> Bernd
>
>
> > An auxiliary variable "struct page **pt_pages" is used in the patch
> > to prepare the 2nd parameter for iov_iter_extract_pages() since
> > iov_iter_get_pages2() uses a different type for the 2nd parameter.
> >
> > Signed-off-by: Lei Huang <lei.huang@xxxxxxxxxxxxxxx>
> > ---
> >   fs/fuse/file.c | 13 ++++++++-----
> >   1 file changed, 8 insertions(+), 5 deletions(-)
> >
> > diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> > index bc41152..715de3b 100644
> > --- a/fs/fuse/file.c
> > +++ b/fs/fuse/file.c
> > @@ -670,7 +670,7 @@ static void fuse_release_user_pages(struct 
> > fuse_args_pages *ap,
> >       for (i = 0; i < ap->num_pages; i++) {
> >           if (should_dirty)
> >               set_page_dirty_lock(ap->pages[i]);
> > -        put_page(ap->pages[i]);
> > +        unpin_user_page(ap->pages[i]);
> >       }
> >   }
> > @@ -1428,10 +1428,13 @@ static int fuse_get_user_pages(struct 
> > fuse_args_pages *ap, struct iov_iter *ii,
> >       while (nbytes < *nbytesp && ap->num_pages < max_pages) {
> >           unsigned npages;
> >           size_t start;
> > -        ret = iov_iter_get_pages2(ii, &ap->pages[ap->num_pages],
> > -                    *nbytesp - nbytes,
> > -                    max_pages - ap->num_pages,
> > -                    &start);
> > +        struct page **pt_pages;
> > +
> > +        pt_pages = &ap->pages[ap->num_pages];
> > +        ret = iov_iter_extract_pages(ii, &pt_pages,
> > +                         *nbytesp - nbytes,
> > +                         max_pages - ap->num_pages,
> > +                         0, &start);
> >           if (ret < 0)
> >               break;