Re: POSIX violation by writeback error

[Vito Caputo]

On Tue, Sep 04, 2018 at 04:18:18PM -0400, Jeff Layton wrote:
> On Tue, 2018-09-04 at 14:54 -0400, J. Bruce Fields wrote:
> > On Tue, Sep 04, 2018 at 06:23:48PM +0200, Rogier Wolff wrote:
> > > On Tue, Sep 04, 2018 at 12:12:03PM -0400, J. Bruce Fields wrote:
> > > > Well, I think the point was that in the above examples you'd prefer that
> > > > the read just fail--no need to keep the data.  A bit marking the file
> > > > (or even the entire filesystem) unreadable would satisfy posix, I guess.
> > > > Whether that's practical, I don't know.
> > > 
> > > When you would do it like that (mark the whole filesystem as "in
> > > error") things go from bad to worse even faster. The Linux kernel 
> > > tries to keep the system up even in the face of errors. 
> > > 
> > > With that suggestion, having one application run into a writeback
> > > error would effectively crash the whole system because the filesystem
> > > may be the root filesystem and stuff like "sshd" that you need to
> > > diagnose the problem needs to be read from the disk.... 
> > 
> > Well, the absolutist position on posix compliance here would be that a
> > crash is still preferable to returning the wrong data.  And for the
> > cases çæå gives, that sounds right?  Maybe it's the wrong balance in
> > general, I don't know.  And we do already have filesystems with
> > panic-on-error options, so if they aren't used maybe then maybe users
> > have already voted against that level of strictness.
> > 
> 
> Yeah, idk. The problem here is that this is squarely in the domain of
> implementation defined behavior. I do think that the current "policy"
> (if you call it that) of what to do after a wb error is weird and wrong.
> What we probably ought to do is start considering how we'd like it to
> behave.
> 
> How about something like this?
> 
> Mark the pages as "uncleanable" after a writeback error. We'll satisfy
> reads from the cached data until someone calls fsync, at which point
> we'd return the error and invalidate the uncleanable pages.
> 
> If no one calls fsync and scrapes the error, we'll hold on to it for as
> long as we can (or up to some predefined limit) and then after that
> we'll invalidate the uncleanable pages and start returning errors on
> reads. If someone eventually calls fsync afterward, we can return to
> normal operation.
> 
> As always though...what about mmap? Would we need to SIGBUS at the point
> where we'd start returning errors on read()?
> 
> Would that approximate the current behavior enough and make sense?
> Implementing it all sounds non-trivial though...
> 

Here's a crazy and potentially stupid idea:

Implement a new class of swap space for backing dirty pages which fail
to write back.  Pages in this space survive reboots, essentially backing
the implicit commitment POSIX establishes in the face of asynchronous
writeback errors.  Rather than evicting these pages as clean, they are
swapped out to the persistent swap.

Administrators then decide if they want to throw some cheap storage at
enabling this coverage, or live with the existing risks.

I think it may be an interesting approach, enabling administrators to
repair primary storage while operating in a degraded mode.  Then once
things are corrected, there would be a way to evict pages from
persistent swap for another shot at writeback to primary storage.

Regards,
Vito Caputo