Re: [PATCH 0/8] ext4: fix endio zeroout corruption by tracking to-be-converted unwritten IOs
From: Zhang Yi
Date: Wed Nov 11 2026 - 22:34:12 EST
On 11/3/2026 2:26 AM, Ojaswin Mujoo wrote:
> There is a data corruption in the zeroout fallback
> ext4_split_extent_zeroout() when we encounter an error like ENOSPC
> while trying to perform endio extent splitting.
>
> The issue was discovered by Fedora coreos community while using 1kb
> blocks size on a 64kb pagesize arch like powerpc or aarch64 and the
> discussion can be found at [1].
>
> ** The Issue **
>
> The issue can be encountered as follows:
>
> 1. Make an FS with no reserved to root reserved blocks and fill it
> completely.
> 2. Delete a file so that ext4_sb_info thinks there is free space but
> mballoc doesnt have any (until txn is committed).
> 3. Now when endio kicks in it eventually reaches:
>
> ext4_handle_unwritten_extents
> ext4_split_convert_extents
> ext4_split_extent
> ext4_split_extent_at
> ext4_ext_insert_extent
> ext4_new_meta_blocks // extent tree split/grow needs a block
> ext4_mb_new_blocks
> ext4_claim_free_clusters // thinks we have free space but mballoc
> has none until txn commit frees it
> ext4_mb_regular_allocator
> // return -ENOSPC
> // ext4_split_extent fallsback to zeroout
> ext4_split_extent_zeroout
> // We try to zeroout surrounding unwritten ranges. This code has a bug
> where we zero an unwritten extent which could have valid data on disk
> whose endio has not yet converted it
>
> The main issue is that zeroout assumes any unwritten extent is valid to
> zeroout but it could actually be a soon-to-be-converted extent holding
> valid data.
>
Hi Ojaswin,
Thank you for your analysis! This makes sense to me. I apologize for not
having considered this concurrent scenario earlier and for amplifying
the data loss issue.
> ** The fix **
>
> The way this series fixes it is by per-inode tracking the no. of
> unwritten IOs that are submitted but not yet converted
> (ei->i_conv_pending_ios). If we end up in the zeroout fallback and see
> there are such IOs existing in the inode, then we don't zeroout and just
> return back the original error. More design notes can be found in
> subsequent patches.
>
Regarding the solution, I agree that your approach looks like it can fix
this issue, and the idea is fairly straightforward. When zeroing out, if
there is other concurrent I/O, just fall back and return an error.
However, I've been thinking about it over the past few days, hoping to
find another simpler and more elegant solution, but that seems rather
difficult.
First, due to the map range caching mechanism in the iomap writeback
path and for performance reasons, we probably can't go back to splitting
extents when issuing I/O as before. Moreover, the previous approach also
couldn't completely avoid zeroing the entire unwritten extent when the
I/O completes, so the problem already existed before. It just required
concurrency and had a very low probability. Therefore, I agree that we
need to address this problem head-on.
Here's a rough idea I'd like to share:
1. Add an in-core extent status flag, something like
EXTENT_STATUS_PENDING_WRITTEN. Like EXTENT_STATUS_DELAYED, these
entries would be must-keep. The idea is to mark ranges for which
we've submitted I/O over an unwritten extent but haven't done the
conversion yet. It would just be an extra flag on top of
EXTENT_STATUS_UNWRITTEN; the zeroout paths would be the only thing
looking at it, and everywhere else the entry would still behave like
a normal unwritten extent. We'd have to drop the flag whenever the
entry changes state, and also when the I/O or the conversion fails.
2. When submitting I/O, once we know the actual range being submitted,
we'd split the existing unwritten extent status entry and add a
pending-written entry for that subrange. Then, when the I/O
completes, or when fallocate and similar paths want to split an
unwritten extent, we could check the extent status tree for the range
we're about to zero; if there's a pending-written entry there, we'd
just skip the zeroout fallback and return the original error.
3. We'd need to set the pending-written entry under i_data_sem. Since
ext4_split_extent_zeroout() already runs under i_data_sem, that
should be enough to avoid the race.
4. One thing to be careful about: a pending-written entry is still
unwritten, but it shouldn't be merged with a normal unwritten entry.
Some advantages I see:
1. We don't need to add anything to struct ext4_inode_info.
2. We don't need a new lock, and we avoid things like
EXT4_GET_BLOCKS_NO_CONV_ACC for paths that don't account for
themselves, as well as walking ioend/iovec at I/O completion just to
keep a count. I think the code would be a bit easier to follow.
3. Random I/O into preallocated files seems pretty common, so tracking
at per-extent (or more precisely, per-range) granularity should give
better exclusion than a per-inode counter and hopefully reduce the
chance of writeback/conversion failures.
A few disadvantages I'm a bit worried about:
1. We'd have to audit all the ext4_es_cache_extent() callers for
conflicts, and all ext4_es_remove_extent()/ext4_es_insert_extent()
call sites, to make sure pending-written entries don't get dropped or
overwritten. I'm not sure yet whether this is definitely feasible.
2. It changes the core semantics of the extent status tree, so it's
quite a bit more code and work, and probably not something for
stable. I see it more as a long-term fix.
What do you think?
Thanks,
Yi.
> For direct IO, ext4_iomap_begin() is called once per mapping and has no
> way to account something only once per dio. So we first fix it in a
> simple way by relying on inode->i_dio_count which is already
> incremented/decremented once per dio. The last patch optimizes this by
> moving dio accounting to i_conv_pending_ios as well, by using a custom
> ext4_iomap_next() (needed because we want to increment once per
> dio). Since iomap_next() infra was added in 7.2+ this can't be
> backported so kept this approach separate.
>
> There are other approaches considered like completely disabling zeroout
> (not ideal as it's a useful fallback) or tracking the conv_pending ranges
> in extent cache (this is an intrusive change) however the current
> approach was chosen as a middle ground to balance complexity vs
> accuracy of accounting.
>
> ** Backport Notes **
>
> This issue seems to have existed ever since we had the zeroout logic
> however this was never encountered until we moved extent splitting from
> writeback time to endio time in commit:
>
> ea96cb5c4ae3 ("ext4: don't split extent before submitting I/O") (merged
> in 7.0)
>
> which changed the dioread_nolock writeback/dio path to submit I/O
> without pre-splitting extents, deferring the split to endio. This means
> multiple concurrent writes can share a single unwritten extent, and a
> failed split on one of them could zeroout data belonging to another,
> which exposes this issue in the zeroout logic.
>
> Note that the issue in zeroout was always present but the above patch
> just uncovered it. However, for the purpose of backports, I've used this
> patch. Seeing how we never saw this issue before this, hopefully just
> backporting it till 7.0 should be fine.
>
> ** Testing **
>
> I've tested -g quick for various configurations to catch any obvious
> issues or counter leaks, saw no regressions. I'll continue to test for
> auto group. I've also added some new tests to hit this regression which
> I'll post soon.
>
> [1] https://github.com/coreos/fedora-coreos-tracker/issues/2234
>
> Thoughts and suggestions welcome.
>
> Regards,
> ojaswin
>
> Signed-off-by: Ojaswin Mujoo <ojaswin@xxxxxxxxxxxxx>
> ---
> Ojaswin Mujoo (8):
> ext4: add extra_flags param to ext4_convert_unwritten_extents()
> ext4: fix endio zeroout corruption by tracking unwritten to-be-converted IOs
> ext4: add tracing for i_conv_pending_ios tracking
> ext4: don't zeroout if we have inflight unwritten IOs in non buffered-iomap path
> ext4: don't zeroout if we have inflight direct IOs
> ext4: make sure zero range WRITE_ZEROES uses NO_CONV_ACC flag
> ext4: don't zeroout if we have inflight unwritten IOs in buffered iomap path
> ext4: optimize inflight dio tracking for zeroout
>
> fs/ext4/ext4.h | 29 +++++++--
> fs/ext4/extents-test.c | 2 +
> fs/ext4/extents.c | 117 ++++++++++++++++++++++++++++-------
> fs/ext4/file.c | 14 ++++-
> fs/ext4/inode.c | 146 +++++++++++++++++++++++++++++++++++++++++++-
> fs/ext4/page-io.c | 27 +++++++-
> fs/ext4/super.c | 12 ++++
> include/trace/events/ext4.h | 25 ++++++++
> 8 files changed, 342 insertions(+), 30 deletions(-)
> ---
> base-commit: 8c4f6e3376348fc3162eed86a05b933921c8f9ed
> change-id: 20261023-ext4-endio-zero-bug-ec627ae61f39
>
> Best regards,