Re: [PATCH v8] fuse: add more control over cache invalidation behaviour

[Luis Henriques]

Hi Miklos,

[ adding Laura to CC, something I should have done before ]

On Mon, Mar 10 2025, Miklos Szeredi wrote:

> On Fri, 7 Mar 2025 at 16:31, Luis Henriques <luis@xxxxxxxxxx> wrote:
>
>> Any further feedback on this patch, or is it already OK for being merged?
>
> The patch looks okay.  I have ideas about improving the name, but that can wait.
>
> What I think is still needed is an actual use case with performance numbers.

As requested, I've run some tests on CVMFS using this kernel patch[1].
For reference, I'm also sharing the changes I've done to libfuse[2] and
CVMFS[3] in order to use this new FUSE operation.  The changes to these
two repositories are in a branch named 'wip-notify-inc-epoch'.

As for the details, basically what I've done was to hack the CVMFS loop in
FuseInvalidator::MainInvalidator() so that it would do a single call to
the libfuse operation fuse_lowlevel_notify_increment_epoch() instead of
cycling through the inodes list.  The CVMFS patch is ugly, it just
short-circuiting the loop, but I didn't want to spend any more time with
it at this stage.  The real patch will be slightly more complex in order
to deal with both approaches, in case the NOTIFY_INC_EPOCH isn't
available.

Anyway, my test environment was a small VM, where I have two scenarios: a
small file-system with just a few inodes, and a larger one with around
8000 inodes.  The test approach was to simply mount the filesystem, load
the caches with 'find /mnt' and force a flush using the cvmfs_swissknife
tool, with the 'ingest' command.

[ Disclosure: my test environment actually uses a fork of upstream cvmfs,
  but for the purposes of these tests that shouldn't really make any
  difference. ]

The numbers in the table below represent the average time (tests were run
100 times) it takes to run the MainInvalidator() function.  As expected,
using the NOTIFY_INC_EPOCH is much faster, as it's a single operation, a
single call into FUSE.  Using the NOTIFY_INVAL_* is much more expensive --
it requires calling into the kernel several times, depending on the number
of inodes on the list.

|------------------+------------------+----------------|
|                  | small filesystem | "big" fs       |
|                  | (~20 inodes)     | (~8000 inodes) |
|------------------+------------------+----------------|
| NOTIFY_INVAL_*   | 330 us           | 4300 us        |
| NOTIFY_INC_EPOCH | 40 us            | 45 us          |
|------------------+------------------+----------------|

Hopefully these results help answering Miklos questions regarding the
cvmfs use-case.

[1] https://lore.kernel.org/all/20250226091451.11899-1-luis@xxxxxxxxxx/
[2] https://github.com/luis-henrix/libfuse
[3] https://github.com/luis-henrix/cvmfs

Cheers,
-- 
Luís