Re: [REGRESSION] ext4: silent data loss when a split at I/O completion gets ENOSPC

From: Ojaswin Mujoo

Date: Wed Nov 18 2026 - 13:35:02 EST


On Wed, Nov 18, 2026 at 06:48:50PM +0800, Zhang Yi wrote:
> On 11/18/2026 3:55 PM, Ojaswin Mujoo wrote:
> > On Tue, Nov 17, 2026 at 07:37:10PM +0800, Zhang Yi wrote:
> >> On 11/16/2026 10:27 AM, Jose Fernandez (Anthropic) wrote:
> >>> Ojaswin's cover letter [1] explains how, since commit ea96cb5c4ae3
> >>> ("ext4: don't split extent before submitting I/O"), the split of an
> >>> unwritten extent at I/O completion can get ENOSPC. [1] stops the zeroout
> >>> from hitting other writes in flight, and [2] reports the failed
> >>> conversion to fsync(). Neither gives the split its block, so data is
> >>> still lost.
> >>>
> >>> LLM agents working with me found that it happens with default options
> >>> on a filesystem that is not full, and ran the tests. write() and fsync()
> >>> return success, and after the pages are evicted part of the file reads
> >>> back as zeros.
> >>>
> >>> Under a load built to run the allocator dry (QEMU without KVM), 42 of
> >>> 72 files lost data at ea96cb5c4ae3, none of 72 at its parent and none
> >>> of 48 on v6.18.63. On mainline at 23c62304dde2 32 of 72 did. With [1]
> >>> rebased on it 42 of 72 still did, now as failed conversions in dmesg.
> >>> With [1] and [2] fsync() failed with ENOSPC on 47 of 72, and the other
> >>> 25 read back intact.
> >>>
> >>
> >> Hi Jose!
> >>
> >> I'm sorry for causing this regression. The cause of this problem is that
> >> the transaction freeing the file's space has not yet been committed and
> >> thus cannot be returned to the mb allocator, so the extent cannot be
> >> split at endio time to allocate space for metadata. This is easily
> >> reproduced in scenarios with frequent file creation and deletion, even
> >> though there is still plenty of free space on disk.
> >>
> >> I think the solution should be based on Ojaswin's counting approach in
> >> [1] or the PENDING_CONVERSION solution we discussed, and then, after
> >> extent splitting fails, try to flush the journal and retry. My idea is
> >> roughly as follows:
> >>
> >> 1. For the iomap buffered I/O path and the DIO path, this is relatively
> >> simple. We just need to add journal flush and retry logic in
> >> ext4_iomap_finish_ioend() and ext4_dio_write_end_io().
> >>
> >> 2. For the traditional bh buffered I/O path, we cannot flush the journal
> >> in ext4_convert_unwritten_io_end_vec(), because this may lead to an
> >> ABBA deadlock with data=ordered mode. The journal process needs to
> >> wait for ordered folio writeback, but this folio may contain some
> >> ranges that need to be converted, and if we flush the journal after
> >> the conversion fails, it will form a mutual dependency with journal
> >> flushing. Therefore, we have to fall back to splitting before issuing
> >> I/O, and when splitting fails, flush the journal and retry via
> >> jbd2_journal_force_commit_nested() in ext4_do_writepages().
> >>
> >> 3. Split the workqueues used at the end of iomap and bh writeback to
> >> avoid deadlocks. Because the filesystem may simultaneously have
> >> different inodes going through the bh path and the iomap path, if the
> >> iomap completion worker is flushing the journal, it may need to wait
> >> for the folio of another inode going through the bh path to finish
> >> conversion, and that conversion worker still needs to be handled on
> >> the same wq, thus causing a deadlock.
> >>
> >> The current bh buffered I/O handling flow cannot be unified with the
> >> other two, which is indeed not ideal. However, once data=ordered mode is
> >> sunset in the future, we should be able to unify them. What do you all
> >> think of this solution? Any better ideas?
> >>
> >> Thanks,
> >> Yi.
> >
> > Hey Zhang, thanks for the poc, it looks largely okay. I agree that this
> > seems to be the right way forward although its unfortunate to have to
> > revert back the unwrit to unwrit split. Hopefully we can move completely
> > switch to iomap soon enough.
> >
> > On the extent status cache for zeroout approach, i'm currently fixing a
> > few bs < ps bugs I'm seeing in it and would clean it up and post it as
> > v2. My idea is to keep the first patch disabling zeroout and rest of
> > them implementing the es changes and renabling it.
>
> Hi Ojaswin,
>
> Yeah, this makes sense to me. Looking forward to your v2. I'll re-adapt
> this POC based on your patch afterwards. :)
>
> > We can just backport
> > the first patch. Along with your poc fixes here also backported to 7.0
> > i believe we should be good to go.
> >
>
> Sorry, I don't quite understand. As far as I know, 7.0 is already EOL, so
> why does this still need to be backported to this version?

Ahh right I just noticed. There's no longterm kernel since 6.18. Cool
then, we don't even need a backport for now which is nice.

Regards,
ojaswin
>
> Thanks,
> Yi.
>
>