Re: [PATCH v8] fuse: add more control over cache invalidation behaviour

[Laura Promberger]

Hello Miklos, Luis,

I tested Luis NOTIFY_INC_EPOCH patch (kernel, libfuse, cvmfs) on RHEL9 and can confirm that in combination with your fix to the symlink truncate it solves all the problem we had with cvmfs when applying a new revision and at the same time hammering a symlink with readlink() that would change its target. ( https://github.com/cvmfs/cvmfs/issues/3626 )

With those two patches we no longer end up with corrupted symlinks or get stuck on an old revision.
(old revision was possible because the kernel started caching the old one again during the update due to the high access rate and the asynchronous evict of inodes)

As such we would be very happy if this patch could be accepted.

Have a nice weekend
Laura


________________________________________
From: Luis Henriques <luis@xxxxxxxxxx>
Sent: Monday, March 17, 2025 12:28
To: Miklos Szeredi <miklos@xxxxxxxxxx>
Cc: Laura Promberger <laura.promberger@xxxxxxx>; Bernd Schubert <bschubert@xxxxxxx>; Dave Chinner <david@xxxxxxxxxxxxx>; Matt Harvey <mharvey@xxxxxxxxxxxxxxx>; linux-fsdevel@xxxxxxxxxxxxxxx <linux-fsdevel@xxxxxxxxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx <linux-kernel@xxxxxxxxxxxxxxx>
Subject: Re: [PATCH v8] fuse: add more control over cache invalidation behaviour
 
Hi Miklos,

[ adding Laura to CC, something I should have done before ]

On Mon, Mar 10 2025, Miklos Szeredi wrote:

> On Fri, 7 Mar 2025 at 16:31, Luis Henriques <luis@xxxxxxxxxx> wrote:
>
>> Any further feedback on this patch, or is it already OK for being merged?
>
> The patch looks okay.  I have ideas about improving the name, but that can wait.
>
> What I think is still needed is an actual use case with performance numbers.

As requested, I've run some tests on CVMFS using this kernel patch[1].
For reference, I'm also sharing the changes I've done to libfuse[2] and
CVMFS[3] in order to use this new FUSE operation.  The changes to these
two repositories are in a branch named 'wip-notify-inc-epoch'.

As for the details, basically what I've done was to hack the CVMFS loop in
FuseInvalidator::MainInvalidator() so that it would do a single call to
the libfuse operation fuse_lowlevel_notify_increment_epoch() instead of
cycling through the inodes list.  The CVMFS patch is ugly, it just
short-circuiting the loop, but I didn't want to spend any more time with
it at this stage.  The real patch will be slightly more complex in order
to deal with both approaches, in case the NOTIFY_INC_EPOCH isn't
available.

Anyway, my test environment was a small VM, where I have two scenarios: a
small file-system with just a few inodes, and a larger one with around
8000 inodes.  The test approach was to simply mount the filesystem, load
the caches with 'find /mnt' and force a flush using the cvmfs_swissknife
tool, with the 'ingest' command.

[ Disclosure: my test environment actually uses a fork of upstream cvmfs,
  but for the purposes of these tests that shouldn't really make any
  difference. ]

The numbers in the table below represent the average time (tests were run
100 times) it takes to run the MainInvalidator() function.  As expected,
using the NOTIFY_INC_EPOCH is much faster, as it's a single operation, a
single call into FUSE.  Using the NOTIFY_INVAL_* is much more expensive --
it requires calling into the kernel several times, depending on the number
of inodes on the list.

|------------------+------------------+----------------|
|                  | small filesystem | "big" fs       |
|                  | (~20 inodes)     | (~8000 inodes) |
|------------------+------------------+----------------|
| NOTIFY_INVAL_*   | 330 us           | 4300 us        |
| NOTIFY_INC_EPOCH | 40 us            | 45 us          |
|------------------+------------------+----------------|

Hopefully these results help answering Miklos questions regarding the
cvmfs use-case.

[1] https://lore.kernel.org/all/20250226091451.11899-1-luis@xxxxxxxxxx/
[2] https://github.com/luis-henrix/libfuse
[3] https://github.com/luis-henrix/cvmfs

Cheers,
--
Luís