Re: [man-pages RFC PATCH v4] statx, inode: document the new STATX_INO_VERSION field

[J. Bruce Fields]

On Thu, Sep 08, 2022 at 11:44:33AM -0400, Jeff Layton wrote:
> On Thu, 2022-09-08 at 11:21 -0400, Theodore Ts'o wrote:
> > On Thu, Sep 08, 2022 at 10:33:26AM +0200, Jan Kara wrote:
> > > It boils down to the fact that we don't want to call mark_inode_dirty()
> > > from IOCB_NOWAIT path because for lots of filesystems that means journal
> > > operation and there are high chances that may block.
> > > 
> > > Presumably we could treat inode dirtying after i_version change similarly
> > > to how we handle timestamp updates with lazytime mount option (i.e., not
> > > dirty the inode immediately but only with a delay) but then the time window
> > > for i_version inconsistencies due to a crash would be much larger.
> > 
> > Perhaps this is a radical suggestion, but there seems to be a lot of
> > the problems which are due to the concern "what if the file system
> > crashes" (and so we need to worry about making sure that any
> > increments to i_version MUST be persisted after it is incremented).
> > 
> > Well, if we assume that unclean shutdowns are rare, then perhaps we
> > shouldn't be optimizing for that case.  So.... what if a file system
> > had a counter which got incremented each time its journal is replayed
> > representing an unclean shutdown.  That shouldn't happen often, but if
> > it does, there might be any number of i_version updates that may have
> > gotten lost.  So in that case, the NFS client should invalidate all of
> > its caches.
> > 
> > If the i_version field was large enough, we could just prefix the
> > "unclean shutdown counter" with the existing i_version number when it
> > is sent over the NFS protocol to the client.  But if that field is too
> > small, and if (as I understand things) NFS just needs to know when
> > i_version is different, we could just simply hash the "unclean
> > shtudown counter" with the inode's "i_version counter", and let that
> > be the version which is sent from the NFS client to the server.
> > 
> > If we could do that, then it doesn't become critical that every single
> > i_version bump has to be persisted to disk, and we could treat it like
> > a lazytime update; it's guaranteed to updated when we do an clean
> > unmount of the file system (and when the file system is frozen), but
> > on a crash, there is no guaranteee that all i_version bumps will be
> > persisted, but we do have this "unclean shutdown" counter to deal with
> > that case.
> > 
> > Would this make life easier for folks?
> > 
> > 						- Ted
> 
> Thanks for chiming in, Ted. That's part of the problem, but we're
> actually not too worried about that case:
> 
> nfsd mixes the ctime in with i_version, so you'd have to crash+clock
> jump backward by juuuust enough to allow you to get the i_version and
> ctime into a state it was before the crash, but with different data.
> We're assuming that that is difficult to achieve in practice.

But a change in the clock could still cause our returned change
attribute to go backwards (even without a crash).  Not sure how to
evaluate the risk, but it was enough that Trond hasn't been comfortable
with nfsd advertising NFS4_CHANGE_TYPE_IS_MONOTONIC.

Ted's idea would be sufficient to allow us to turn that flag on, which I
think allows some client-side optimizations.

> The issue with a reboot counter (or similar) is that on an unclean crash
> the NFS client would end up invalidating every inode in the cache, as
> all of the i_versions would change. That's probably excessive.

But if we use the crash counter on write instead of read, we don't
invalidate caches unnecessarily.  And I think the monotonicity would
still be close enough for our purposes?

> The bigger issue (at the moment) is atomicity: when we fetch an
> i_version, the natural inclination is to associate that with the state
> of the inode at some point in time, so we need this to be updated
> atomically with certain other attributes of the inode. That's the part
> I'm trying to sort through at the moment.

That may be, but I still suspect the crash counter would help.

--b.