Re: [REGRESSION] ext4: silent data loss when a split at I/O completion gets ENOSPC
From: Zhang Yi
Date: Wed Nov 18 2026 - 05:49:32 EST
On 11/18/2026 3:55 PM, Ojaswin Mujoo wrote:
> On Tue, Nov 17, 2026 at 07:37:10PM +0800, Zhang Yi wrote:
>> On 11/16/2026 10:27 AM, Jose Fernandez (Anthropic) wrote:
>>> Ojaswin's cover letter [1] explains how, since commit ea96cb5c4ae3
>>> ("ext4: don't split extent before submitting I/O"), the split of an
>>> unwritten extent at I/O completion can get ENOSPC. [1] stops the zeroout
>>> from hitting other writes in flight, and [2] reports the failed
>>> conversion to fsync(). Neither gives the split its block, so data is
>>> still lost.
>>>
>>> LLM agents working with me found that it happens with default options
>>> on a filesystem that is not full, and ran the tests. write() and fsync()
>>> return success, and after the pages are evicted part of the file reads
>>> back as zeros.
>>>
>>> Under a load built to run the allocator dry (QEMU without KVM), 42 of
>>> 72 files lost data at ea96cb5c4ae3, none of 72 at its parent and none
>>> of 48 on v6.18.63. On mainline at 23c62304dde2 32 of 72 did. With [1]
>>> rebased on it 42 of 72 still did, now as failed conversions in dmesg.
>>> With [1] and [2] fsync() failed with ENOSPC on 47 of 72, and the other
>>> 25 read back intact.
>>>
>>
>> Hi Jose!
>>
>> I'm sorry for causing this regression. The cause of this problem is that
>> the transaction freeing the file's space has not yet been committed and
>> thus cannot be returned to the mb allocator, so the extent cannot be
>> split at endio time to allocate space for metadata. This is easily
>> reproduced in scenarios with frequent file creation and deletion, even
>> though there is still plenty of free space on disk.
>>
>> I think the solution should be based on Ojaswin's counting approach in
>> [1] or the PENDING_CONVERSION solution we discussed, and then, after
>> extent splitting fails, try to flush the journal and retry. My idea is
>> roughly as follows:
>>
>> 1. For the iomap buffered I/O path and the DIO path, this is relatively
>> simple. We just need to add journal flush and retry logic in
>> ext4_iomap_finish_ioend() and ext4_dio_write_end_io().
>>
>> 2. For the traditional bh buffered I/O path, we cannot flush the journal
>> in ext4_convert_unwritten_io_end_vec(), because this may lead to an
>> ABBA deadlock with data=ordered mode. The journal process needs to
>> wait for ordered folio writeback, but this folio may contain some
>> ranges that need to be converted, and if we flush the journal after
>> the conversion fails, it will form a mutual dependency with journal
>> flushing. Therefore, we have to fall back to splitting before issuing
>> I/O, and when splitting fails, flush the journal and retry via
>> jbd2_journal_force_commit_nested() in ext4_do_writepages().
>>
>> 3. Split the workqueues used at the end of iomap and bh writeback to
>> avoid deadlocks. Because the filesystem may simultaneously have
>> different inodes going through the bh path and the iomap path, if the
>> iomap completion worker is flushing the journal, it may need to wait
>> for the folio of another inode going through the bh path to finish
>> conversion, and that conversion worker still needs to be handled on
>> the same wq, thus causing a deadlock.
>>
>> The current bh buffered I/O handling flow cannot be unified with the
>> other two, which is indeed not ideal. However, once data=ordered mode is
>> sunset in the future, we should be able to unify them. What do you all
>> think of this solution? Any better ideas?
>>
>> Thanks,
>> Yi.
>
> Hey Zhang, thanks for the poc, it looks largely okay. I agree that this
> seems to be the right way forward although its unfortunate to have to
> revert back the unwrit to unwrit split. Hopefully we can move completely
> switch to iomap soon enough.
>
> On the extent status cache for zeroout approach, i'm currently fixing a
> few bs < ps bugs I'm seeing in it and would clean it up and post it as
> v2. My idea is to keep the first patch disabling zeroout and rest of
> them implementing the es changes and renabling it.
Hi Ojaswin,
Yeah, this makes sense to me. Looking forward to your v2. I'll re-adapt
this POC based on your patch afterwards. :)
> We can just backport
> the first patch. Along with your poc fixes here also backported to 7.0
> i believe we should be good to go.
>
Sorry, I don't quite understand. As far as I know, 7.0 is already EOL, so
why does this still need to be backported to this version?
Thanks,
Yi.