Re: 9p caching with cache=loose and cache=fscache

[Eric Van Hensbergen]

Sorry, took a bit to unstack from day job, but while going through the
patch queue I remembered I still had some questions to answer here.

On Fri, Mar 17, 2023 at 12:01 PM Luis Chamberlain <mcgrof@xxxxxxxxxx> wrote:
>
> On Sun, Mar 12, 2023 at 01:22:34PM -0500, Eric Van Hensbergen wrote:
> > I was looking at kdevops the other day - cool stuff.  Was trying to
> > figure out how we could do v9fs CI with it.
>
> Happy to help any questions you may have about it!
>
> > Both cache=loose and cache=fscache currently don't validate via host.
>
> What does this mean exactly?
>

That's a good question - I guess the answer is "by design" they don't
do anything special to check that the cache is up to date with the
host.  That being said, there are several places in the code where the
cache will be invalidated (in some cases even if its up to date with
the host, yes, those are likely bugs).  Many of the invalidations that
are present are overly conservative.  The key missing one is
v9fs_lookup_revalidate which currently fills missing inode data but
should potentially also check to see if anything in the file changed
and react accordingly.  The other thing is that we aren't always using
the caches when we should, if you look at the traffic even when
everything should be perfectly cached in fscache we are getting data
and metadata from the wire -- those are mostly bugs in the cache
implementation that I'm trying to go through and fix now.

>From the 9p perspective, we should be looking at qid.version values
returned from the host and matching them to our internal notion of
version.  If the versions don't match we should be invalidating
caches.  qid.versions get returned on open, lookup, and most meta-data
operations so there's lots of opportunities there.  There are still
some issues with this approach, namely that not all servers populate
qid.version and right now the ones that do use a hashed timestamp
(since qid.version is only 32-bits).  This probably covers most bases,
but its not ideal -- so currrently thinking through whether we do a
further extension of the protocol or use some other mechanism.
There's a possibility of using a full getattr every so often to
back-up qid.version validation, but if we want truly tight coherence
(not just open to close) then we need to open up some sort of back
channel for invalidates from the server -- but I think the qid based
invalidation probably gets us most of what we need so going to start
with that.

> Right now a host with debian 6.0.0-6-amd64 certainly does not seem to push
> out changes to 9p clients on VMs but Josef informs me that with 6.2-rc8
> he did see the changes propagate.

I did tighten up some of the invalidation in the last round of
patches, however these are likely more on the overly conservative side
versus doing the right thing -- however, its really not at the point
where you can rely on it.  If consistency is something you care about,
I'd suggest cache=none until you can get cache=readahead.

>
> Do none of the existing 9p cache modes not support open-to-close policies
> at all?
>

not specifically open-to-close, loose supports file and dir caching
but without consistency, it might be tempting to try cache=mmap to see
if it gets you closer, but my frame of reference is more the current
patches versus the old code so not sure it would buy you anything.

>
> Right now the cache mode used is cache=loose as that's the default,
> what do you recommend for a typical kernel development environemnt?
>

As I said, if you are interactively changing things I think you'd want
to go for cache=none for now (as painful as it is).  I have fixed what
I hope to be my last bug with the new patch series so it should be
going into linux-next today.

      -eric