Re: [PATCH 00/33] Network fs helper library & fscache kiocb API [ver #3]

From: Jeff Layton
Date: Mon Feb 15 2021 - 13:08:12 EST


On Mon, 2021-02-15 at 15:44 +0000, David Howells wrote:
> Here's a set of patches to do two things:
>
>  (1) Add a helper library to handle the new VM readahead interface. This
>      is intended to be used unconditionally by the filesystem (whether or
>      not caching is enabled) and provides a common framework for doing
>      caching, transparent huge pages and, in the future, possibly fscrypt
>      and read bandwidth maximisation. It also allows the netfs and the
>      cache to align, expand and slice up a read request from the VM in
>      various ways; the netfs need only provide a function to read a stretch
>      of data to the pagecache and the helper takes care of the rest.
>
>  (2) Add an alternative fscache/cachfiles I/O API that uses the kiocb
>      facility to do async DIO to transfer data to/from the netfs's pages,
>      rather than using readpage with wait queue snooping on one side and
>      vfs_write() on the other. It also uses less memory, since it doesn't
>      do buffered I/O on the backing file.
>
>      Note that this uses SEEK_HOLE/SEEK_DATA to locate the data available
>      to be read from the cache. Whilst this is an improvement from the
>      bmap interface, it still has a problem with regard to a modern
>      extent-based filesystem inserting or removing bridging blocks of
>      zeros. Fixing that requires a much greater overhaul.
>
> This is a step towards overhauling the fscache API. The change is opt-in
> on the part of the network filesystem. A netfs should not try to mix the
> old and the new API because of conflicting ways of handling pages and the
> PG_fscache page flag and because it would be mixing DIO with buffered I/O.
> Further, the helper library can't be used with the old API.
>
> This does not change any of the fscache cookie handling APIs or the way
> invalidation is done.
>
> In the near term, I intend to deprecate and remove the old I/O API
> (fscache_allocate_page{,s}(), fscache_read_or_alloc_page{,s}(),
> fscache_write_page() and fscache_uncache_page()) and eventually replace
> most of fscache/cachefiles with something simpler and easier to follow.
>
> The patchset contains five parts:
>
>  (1) Some helper patches, including provision of an ITER_XARRAY iov
>      iterator and a function to do readahead expansion.
>
>  (2) Patches to add the netfs helper library.
>
>  (3) A patch to add the fscache/cachefiles kiocb API.
>
>  (4) Patches to add support in AFS for this.
>
>  (5) Patches from Jeff Layton to add support in Ceph for this.
>
> Dave Wysochanski also has patches for NFS for this, though they're not
> included on this branch as there's an issue with PNFS.
>
> With this, AFS without a cache passes all expected xfstests; with a cache,
> there's an extra failure, but that's also there before these patches.
> Fixing that probably requires a greater overhaul. Ceph and NFS also pass
> the expected tests.
>
> These patches can be found also on:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=fscache-netfs-lib
>
> For diffing reference, the tag for the 9th Feb pull request is
> fscache-ioapi-20210203 and can be found in the same repository.
>
>
>
> Changes
> =======
>
>  (v3) Rolled in the bug fixes.
>
>       Adjusted the functions that unlock and wait for PG_fscache according
>       to Linus's suggestion.
>
>       Hold a ref on a page when PG_fscache is set as per Linus's
>       suggestion.
>
>       Dropped NFS support and added Ceph support.
>
>  (v2) Fixed some bugs and added NFS support.
>
>
> References
> ==========
>
> These patches have been published for review before, firstly as part of a
> larger set:
>
> Link: https://lore.kernel.org/linux-fsdevel/158861203563.340223.7585359869938129395.stgit@xxxxxxxxxxxxxxxxxxxxxx/
>
> Link: https://lore.kernel.org/linux-fsdevel/159465766378.1376105.11619976251039287525.stgit@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-fsdevel/159465784033.1376674.18106463693989811037.stgit@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-fsdevel/159465821598.1377938.2046362270225008168.stgit@xxxxxxxxxxxxxxxxxxxxxx/
>
> Link: https://lore.kernel.org/linux-fsdevel/160588455242.3465195.3214733858273019178.stgit@xxxxxxxxxxxxxxxxxxxxxx/
>
> Then as a cut-down set:
>
> Link: https://lore.kernel.org/linux-fsdevel/161118128472.1232039.11746799833066425131.stgit@xxxxxxxxxxxxxxxxxxxxxx/
>
> Link: https://lore.kernel.org/linux-fsdevel/161161025063.2537118.2009249444682241405.stgit@xxxxxxxxxxxxxxxxxxxxxx/
>
>
> Proposals/information about the design has been published here:
>
> Link: https://lore.kernel.org/lkml/24942.1573667720@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-fsdevel/2758811.1610621106@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-fsdevel/1441311.1598547738@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-fsdevel/160655.1611012999@xxxxxxxxxxxxxxxxxxxxxx/
>
> And requests for information:
>
> Link: https://lore.kernel.org/linux-fsdevel/3326.1579019665@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-fsdevel/4467.1579020509@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-fsdevel/3577430.1579705075@xxxxxxxxxxxxxxxxxxxxxx/
>
> The NFS parts, though not included here, have been tested by someone who's
> using fscache in production:
>
> Link: https://listman.redhat.com/archives/linux-cachefs/2020-December/msg00000.html
>
> I've posted partial patches to try and help 9p and cifs along:
>
> Link: https://lore.kernel.org/linux-fsdevel/1514086.1605697347@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-cifs/1794123.1605713481@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-fsdevel/241017.1612263863@xxxxxxxxxxxxxxxxxxxxxx/
> Link: https://lore.kernel.org/linux-cifs/270998.1612265397@xxxxxxxxxxxxxxxxxxxxxx/
>
> David
> ---
> David Howells (27):
>       iov_iter: Add ITER_XARRAY
>       mm: Add an unlock function for PG_private_2/PG_fscache
>       mm: Implement readahead_control pageset expansion
>       vfs: Export rw_verify_area() for use by cachefiles
>       netfs: Make a netfs helper module
>       netfs, mm: Move PG_fscache helper funcs to linux/netfs.h
>       netfs, mm: Add unlock_page_fscache() and wait_on_page_fscache()
>       netfs: Provide readahead and readpage netfs helpers
>       netfs: Add tracepoints
>       netfs: Gather stats
>       netfs: Add write_begin helper
>       netfs: Define an interface to talk to a cache
>       netfs: Hold a ref on a page when PG_private_2 is set
>       fscache, cachefiles: Add alternate API to use kiocb for read/write to cache
>       afs: Disable use of the fscache I/O routines
>       afs: Pass page into dirty region helpers to provide THP size
>       afs: Print the operation debug_id when logging an unexpected data version
>       afs: Move key to afs_read struct
>       afs: Don't truncate iter during data fetch
>       afs: Log remote unmarshalling errors
>       afs: Set up the iov_iter before calling afs_extract_data()
>       afs: Use ITER_XARRAY for writing
>       afs: Wait on PG_fscache before modifying/releasing a page
>       afs: Extract writeback extension into its own function
>       afs: Prepare for use of THPs
>       afs: Use the fs operation ops to handle FetchData completion
>       afs: Use new fscache read helper API
>
> Jeff Layton (6):
>       ceph: disable old fscache readpage handling
>       ceph: rework PageFsCache handling
>       ceph: fix fscache invalidation
>       ceph: convert readpage to fscache read helper
>       ceph: plug write_begin into read helper
>       ceph: convert ceph_readpages to ceph_readahead
>
>
>  fs/Kconfig | 1 +
>  fs/Makefile | 1 +
>  fs/afs/Kconfig | 1 +
>  fs/afs/dir.c | 225 ++++---
>  fs/afs/file.c | 470 ++++---------
>  fs/afs/fs_operation.c | 4 +-
>  fs/afs/fsclient.c | 108 +--
>  fs/afs/inode.c | 7 +-
>  fs/afs/internal.h | 58 +-
>  fs/afs/rxrpc.c | 150 ++---
>  fs/afs/write.c | 610 +++++++++--------
>  fs/afs/yfsclient.c | 82 +--
>  fs/cachefiles/Makefile | 1 +
>  fs/cachefiles/interface.c | 5 +-
>  fs/cachefiles/internal.h | 9 +
>  fs/cachefiles/rdwr2.c | 412 ++++++++++++
>  fs/ceph/Kconfig | 1 +
>  fs/ceph/addr.c | 535 ++++++---------
>  fs/ceph/cache.c | 125 ----
>  fs/ceph/cache.h | 101 +--
>  fs/ceph/caps.c | 10 +-
>  fs/ceph/inode.c | 1 +
>  fs/ceph/super.h | 1 +
>  fs/fscache/Kconfig | 1 +
>  fs/fscache/Makefile | 3 +-
>  fs/fscache/internal.h | 3 +
>  fs/fscache/page.c | 2 +-
>  fs/fscache/page2.c | 117 ++++
>  fs/fscache/stats.c | 1 +
>  fs/internal.h | 5 -
>  fs/netfs/Kconfig | 23 +
>  fs/netfs/Makefile | 5 +
>  fs/netfs/internal.h | 97 +++
>  fs/netfs/read_helper.c | 1169 +++++++++++++++++++++++++++++++++
>  fs/netfs/stats.c | 59 ++
>  fs/read_write.c | 1 +
>  include/linux/fs.h | 1 +
>  include/linux/fscache-cache.h | 4 +
>  include/linux/fscache.h | 40 +-
>  include/linux/netfs.h | 195 ++++++
>  include/linux/pagemap.h | 3 +
>  include/net/af_rxrpc.h | 2 +-
>  include/trace/events/afs.h | 74 +--
>  include/trace/events/netfs.h | 201 ++++++
>  mm/filemap.c | 20 +
>  mm/readahead.c | 70 ++
>  net/rxrpc/recvmsg.c | 9 +-
>  47 files changed, 3473 insertions(+), 1550 deletions(-)
>  create mode 100644 fs/cachefiles/rdwr2.c
>  create mode 100644 fs/fscache/page2.c
>  create mode 100644 fs/netfs/Kconfig
>  create mode 100644 fs/netfs/Makefile
>  create mode 100644 fs/netfs/internal.h
>  create mode 100644 fs/netfs/read_helper.c
>  create mode 100644 fs/netfs/stats.c
>  create mode 100644 include/linux/netfs.h
>  create mode 100644 include/trace/events/netfs.h
>
>

Thanks David,

I did an xfstests run on ceph with a kernel based on this and it seemed
to do fine. I'll plan to pull this into the ceph-client/testing branch
and run it through the ceph kclient test harness. There are only a few
differences from the last run we did, so I'm not expecting big changes,
but I'll keep you posted.

--
Jeff Layton <jlayton@xxxxxxxxxx>