Re: [PATCH v2 00/20] md/md-llbitmap: support reshape for RAID10 and RAID5

[Mykola Marzhan]

Hi Kuai,

We ran v2 through a QEMU test campaign while preparing our own llbitmap
patches, which we will rebase on top of this series: raid1/raid5/raid10
over loop devices, base 55b77337bdd0 ("md/raid5: avoid R5_Overlap races
while breaking stripe batches") with and without the series applied,
KASAN kernels for the reshape stress, and per-bit bitmap dumps as the
oracle. Four problems, worst first — details under each number below:

1. raid5: writes above dev_sectors stop being tracked (steady state,
   no reshape involved)                                 [patches 02,12]
2. a stopped and re-assembled reshape resumes with the pre-reshape
   bitmap geometry, and persists it                     [patch 08]
3. discard can mark a chunk unwritten while half of it still holds
   live data (raid1)                                    [patch 10]
4. pre-existing: raid5_bitmap_sector() rounds with round_down() on a
   non-power-of-2 size — bits land in the wrong stripe; fix offered
                                                        [base, patch 12]

Happy to rerun the whole campaign on v3 and, if it comes back clean,
follow up with Tested-by. We'll post the test scripts in-thread if
useful.

1. raid5: writes above dev_sectors stop being tracked

llbitmap_map_layout() (patch 12) compares an array sector against
llbitmap_logical_size(). Outside a reshape llbitmap_reshaping() is
false, so that falls through llbitmap_personality_sync_size() to
llbitmap->sync_size — which raid5_run() set from
mddev->resync_max_sectors = mddev->dev_sectors, a per-device count.
An array sector compared against a device-sector limit: every write
above array sector dev_sectors gets *sectors = 0, and with patch 02's
early return the bitmap is never touched.

On a 4-disk raid5 tracking stops after the first third of the array,
on a 7-disk after the first sixth; a crash would then leave everything
above that boundary out of the post-crash resync. raid1/raid10 are
unaffected (their resync_max_sectors is the array size), and *during*
a reshape the correct bitmap_array_sectors is consulted — the problem
only shows in steady state.

Measured (one 4k write below the limit, one above, count new dirty
bits): patched 4-disk +1/+0, patched 7-disk +1/+0; base +1/+1 both.

Is llbitmap_logical_size() meant to return array sectors
unconditionally (i.e. consult patch 17's bitmap_array_sectors outside
reshape too), or should llbitmap_map_layout() convert before
comparing? The two helpers currently mix coordinate spaces for
raid456.

2. resumed reshape reverts the bitmap geometry

Two things combine:

- md_run() calls md_bitmap_create() before assigning mddev->pers, so
  llbitmap_refresh_reshape() (patch 08) runs with mddev->pers == NULL
  and llbitmap_personality_sync_size() bails out to the pre-reshape
  sync_size from the superblock;
- on a resumed reshape md never calls pers->start_reshape() — with
  reshape_position != MaxSector only check_reshape() runs — so
  bitmap_ops->resize() is never re-invoked either.

llbitmap_reshape_finish() then commits the stale geometry and
llbitmap_update_sb() persists it, so the revert is permanent: pages
beyond the old used_pages are returned zeroed by llbitmap_read_page()
instead of read from disk.

Reproduces in ~90s on raid10 4->6: grow, stop mid-reshape,
re-assemble — llbitmap/metadata chunks read 8128 -> 12192 -> 8128.
mdadm --grow --continue does not help. raid5 disk-add is immune
(device-indexed bitmap, the resize is a no-op), which makes it a
clean control.

Possible directions: re-run llbitmap_refresh_reshape() once
mddev->pers is set, or have the resume path call
bitmap_ops->resize(). Moving md_bitmap_create() after the pers
assignment would also do it, but we did not audit what else depends
on the current order.

3. sub-chunk discard marks a live chunk unwritten

llbitmap_start_discard() computes its first bit with
DIV_ROUND_UP_SECTOR_T(offset, chunksize) precisely so a partially
discarded head chunk is not touched. Patch 10 makes
llbitmap_prepare_range() run llbitmap_encode_range() on every op,
which floors the offset to a chunk boundary first — the DIV_ROUND_UP
has nothing left to round. raid1/raid10 previously had no
bitmap_sector hook, so raw offsets reached start_discard and the
guard worked.

Test, on raid1: write a pattern, then discard [K+0.5 chunks,
K+1.5 chunks) — a range that fully contains no chunk, so ideally zero
chunks change state. Base flips 1 (the tail chunk — pre-existing,
arguably the same class of problem on the other end), patched flips
2. The extra one is chunk K, whose first half still holds the
pattern.

Maybe encode_range should be skipped for discard, or start_discard
should take the raw offset.

4. pre-existing: raid5_bitmap_sector() rounding is wrong for
   non-power-of-2 data-disk counts

	sectors_per_chunk = conf->chunk_sectors *
		(conf->raid_disks - conf->max_degraded);
	start = round_down(start, sectors_per_chunk);
	end = round_up(end, sectors_per_chunk);

round_down()/round_up() are mask-based and require power-of-2
alignment; chunk_sectors is a power of 2, but multiplied by 3, 5, 6
or 7 data disks the product is not. The rounded sector is then not
stripe-aligned and raid5_compute_sector() maps it into a different
stripe: the bits marked and the bits covering the write are disjoint.
Lost bits, not wide bits.

Per-bit dumps, no reshape running: 3 data disks lose 20/20 targeted
writes, 6 data disks 17/17, the 4-data (power-of-2) control 0/20.
Byte-identical on the unpatched base, so not this series' bug — but
patch 12 routes this mapping into the reshape remap path, so it
matters more afterwards.

The fix needs division-based rounding (sector_div(), these are
sector_t). We can send a standalone patch, or you can fold it into
v3 — whichever is easier.

One gap in our coverage: raid10 grow under sustained IO currently
floods KASAN with pre-existing slab-out-of-bounds reports from the
r10bio width mismatch (free_r10bio() / raid10_make_request()) —
reproduced on the unpatched base too; none of the sites is touched by
your series. That is what Chen Cheng's "md/raid10: fix r10bio width
mismatches across reshape" fixes:

  https://lore.kernel.org/linux-raid/20260711100352.425177-1-chencheng@xxxxxxxxx/

Our reproduction data is in that thread; we will re-run the
raid10-grow KASAN leg here once it lands.

Thanks,
Mykola