Re: [PATCH 0/8] md/raid5: scalability and rebuild-path improvements

[Hiroshi Nishida]

The cover quotes the 2.1-3.2x headline; here is the full per-workload data
behind it, in case it helps review.

The same raid456.ko is used in both arms -- only the module parameters
differ at array-create -- so this isolates the *defaults*, not code changes.
Real NVMe SSDs, not a ramdisk.

Setup: GCP n2-standard-32 -- 32 vCPUs = 16 physical cores (a vCPU is one
hyperthread), 2 NUMA nodes, 125 GB RAM, 16x 375 GB local NVMe. Kernel: full
build of the 6-patch series (base 55b77337), non-debug. raid6, 64k chunk,
bitmap=none, --assume-clean. Workloads: a 5-test fio suite (direct=1,
libaio, 30 s each): 4K random write (RMW), DB-mixed 75/25 8K,
high-concurrency 70/30 4K, OLTP 70/30 16K, partial-stripe 8K.

AUTO = all defaults left auto (the hardware-derived values); STOCK = the
historical fixed values (hash 8 / ceiling 32768 / batch 8 / cache 256 /
group_thread_cnt 0). On this host AUTO resolves to group_thread_cnt=8,
stripe_cache_size~3886, 32 hash locks.

1) Wide 16-disk raid6 (14+2), steady state, interleaved N=5 (STOCK and AUTO
   alternate each round so SSD drift cancels; cv 1-8%):

     Workload             STOCK IOPS    AUTO IOPS    AUTO/STOCK
     4K random write          39,147      100,069        2.56x
     DB mixed 75/25 8K        88,040      208,284        2.37x
     High-conc 70/30 4K      122,765      257,952        2.10x
     OLTP 70/30 16K           39,597      100,033        2.53x
     Partial-stripe 8K        20,491       64,511        3.15x

2) Low-end 4-CPU / 8 GB, 6-disk raid6 (4+2), N=3 -- no-regression control.
   4 online CPUs + 8 GB, single node (num_online_cpus()=4 -> gtc=2); the
   cache stays 256 and the hash count stays 8, so AUTO differs from STOCK
   only in group_thread_cnt (2 vs 0):

     Workload             STOCK IOPS    AUTO IOPS    AUTO/STOCK
     4K random write          52,017       81,132        1.56x
     DB mixed                125,043      126,188        1.01x
     High-conc               180,570      217,074        1.20x
     OLTP                     54,308       55,840        1.03x
     Partial-stripe           27,060       42,751        1.58x

   No regression on any workload; modest write gains even here. A
   slower-disk NAS is more device-bound (the gains flatten) but still shows
   no regression, and a box with <=2 online CPUs derives gtc=0 =
   byte-for-byte stock.

The gain is essentially patch 6 (worker groups on by default); patches 2-5
(cache and lock sizing) are throughput-neutral on real SSD and earn their
place as configurability plus sane defaults, not a speed claim. Patch 1 is
the NUMA-sizing prerequisite and does not affect throughput.

Thanks,

2026年7月10日(金) 6:09 Hiroshi Nishida <nishidafmly@xxxxxxxxx>:
>
> Hi Yu Kuai,
>
> Thanks again for the review, and in particular for:
>
> > I can accept make those values configurable, but not direct
> > modifications.
>
> I've reworked the tunables along exactly those lines and will send them as
> a separate, self-contained series, "md/raid5: size stripe-cache and worker
> tuning from the hardware".
>
> Rather than raising any fixed constant, each value now derives a default
> from the hardware and stays overridable:
>
>   - the stripe-cache hash lock count (was a fixed 8) is sized from the CPU
>     count, clamped to 8..32;
>   - the stripe_cache_size ceiling (was a fixed 32768) scales with memory,
>     but never drops below 32768;
>   - the initial stripe_cache_size (was a fixed 256) scales gently with
>     memory, 256..4096;
>   - the default group_thread_cnt (was 0, i.e. single-threaded) is derived
>     from the CPU count (half the online CPUs per NUMA node, capped);
>   - the stripe batch size (was a fixed 8) is exposed as a parameter.
>
> Ahead of those, the series opens with a one-line prerequisite fix:
> alloc_thread_groups() sizes the per-node worker_groups[] array by
> num_possible_nodes() but indexes it by cpu_to_node(), so a sparse NUMA node
> map can index off the end -- reachable once the worker-group default (the
> last patch) is on.  It is Fixes:-tagged and can be taken on its own.
>
> The key point for the consumer-NAS concern you raised: each default only
> rises on hardware that can back it -- the lock and worker counts scale with
> the core count, the cache sizes with RAM -- so a genuinely small system (a
> few cores and a few GB) keeps today's values and footprint. And each one is
> overridable via a module parameter (group_thread_cnt also via its existing
> sysfs attribute), including all the way back to today's behaviour. So nothing is
> imposed; the default simply tracks the machine instead of a constant, and
> a wide array on a large host no longer needs a recompile or manual
> per-array tuning to use the memory and cores it has.
>
> And on performance -- on real NVMe this time, not a ramdisk: on a 16-disk
> raid6 array (a 32-vCPU / 16-core host, steady state, interleaved runs) the
> hardware-derived defaults run 2.1-3.2x the stock defaults (4K random write
> ~39k -> ~100k IOPS), essentially all of it from patch 6 turning on the
> worker groups.  Patches 2-5 (the cache and lock sizing) are
> throughput-neutral on real SSD, as you'd expect -- they earn their place as
> configurability and sane defaults, not a speed claim.  And there is no
> regression at the small end: a 4-CPU box derives 2 workers and is never
> slower on any workload, and a box with two or fewer CPUs derives 0 and is
> byte-for-byte unchanged.
>
> It has been through KASAN + lockdep + DEBUG_LIST on RAID5
> (create/rebuild/scrub, plus the bitmap add/remove that drives the
> lock-all-hash-locks quiesce path), at both the derived defaults and pinned
> values including nr_stripe_hash_locks=32.
>
> The two unrelated parts of the original series -- the type-widening /
> correctness fixes and the resync/recovery dispatch changes -- I'll send as
> their own small series, as discussed.
>
> Thanks,
>
> 2026年7月5日(日) 19:56 yu kuai <yukuai@xxxxxxx>:
> >
> > Hi,
> >
> > 在 2026/6/24 23:54, Hiroshi Nishida 写道:
> > > This series collects small, individually low-risk md/raid5 changes for
> > > large, many-core, many-disk arrays.  Their common theme is reducing
> > > per-stripe and stripe-cache contention, so the benefit appears mainly
> > > when the raid5 stripe-handling worker threads are in use
> > > (group_thread_cnt > 0); at the default group_thread_cnt = 0 (a single
> > > handling thread) the series is essentially neutral.
> > >
> > >   - patches 1-3 remove signed arithmetic from a hot-path divisor, lift an
> > >     arbitrary stripe-cache size cap, and widen a badblock length argument
> > >     that currently truncates large ranges;
> > >   - patch 4 raises NR_STRIPE_HASH_LOCKS (8 -> 32) to spread stripe-hash
> > >     contention on high core-count systems;
> > >   - patches 5 and 8 reduce per-stripe overhead in the resync/recovery
> > >     path and bound the share of the stripe cache a rebuild may hold while
> > >     user I/O is competing;
> > >   - patch 6 allocates each worker group's array on its own NUMA node;
> > >   - patch 7 raises MAX_STRIPE_BATCH (8 -> 32).
> > >
> > > Measured effect, treatment vs baseline, % change in mean IOPS (N=3),
> > > swept over group_thread_cnt (RAID6 4+2, 22-core host, ramdisk members):
> >
> > Testing with ramdisk does serve as a useful reference, but it does not reflect
> > real world usage.
> >
> > >
> > >    workload                       gtc=0   gtc=2   gtc=4   gtc=8
> > >    random 4K write (RMW)          +4.2%   +8.1%  +17.4%   +6.5%
> > >    DB mixed 75/25 8K              +0.4%   +4.2%  +10.3%   +4.7%
> > >    high-concurrency 70/30 4K      +3.9%   +1.2%  +10.0%   +0.2%
> > >    OLTP 70/30 16K                 -0.3%   +4.7%  +10.1%   +9.3%
> > >    partial-stripe write 8K        +1.1%   +4.8%  +11.2%  +14.2%
> >
> > With a quick review I saw many static configurations is changed, I agree
> > these changes can improve arrays with ssd/nvme and a system with large
> > memory available. However, we already tested with hdd and about 8G memory
> > available, these changes will not improve performance at all, with the
> > extra memory overhead.
> >
> > I can accept make those values configurable, but not direct modifications.
> > As validation is required for numerous scenarios. Memory resources are precious
> > especially for most consumer NAS devices.
> >
> > >
> > > At the default single handling thread (group_thread_cnt = 0) the series is
> > > neutral (no regression).  As worker threads are added the gain grows,
> > > peaking broadly around group_thread_cnt = 4 at roughly +10-17% across the
> > > whole mix; at gtc = 8 the write-heavy workloads keep gaining while the
> > > read-heavy high-concurrency case has saturated.  (Per-run cv was <1%
> > > except the random-write test, ~5-9%, from a cold first run.)
> > >
> > > These numbers are on a ramdisk, which removes device latency and so
> > > overstates the CPU-side contention effect relative to a real device;
> > > they show the direction and the group_thread_cnt dependence, not an
> > > absolute speedup.  The stripe-hash/batch patches (4, 7) and the cache cap
> > > (2) drive this; patch 6 only matters on multi-socket systems (not
> > > exercised above) and patches 5/8 act on the resync/recovery path rather
> > > than this steady-state workload.
> > >
> > > Reproduction (stock mdadm + fio):
> > >    mdadm --create /dev/md0 --level=6 --raid-devices=6 --chunk=512 \
> > >          --assume-clean <6 members>
> > >    echo 16384 > /sys/block/md0/md/stripe_cache_size
> > >    echo N     > /sys/block/md0/md/group_thread_cnt      # N = 0,2,4,8
> > >    fio --filename=/dev/md0 --direct=1 --ioengine=libaio --group_reporting \
> > >        --time_based --runtime=15 --name=w <per-workload opts>:
> > >      random write   : --rw=randwrite              --bs=4k  --numjobs=4  --iodepth=32
> > >      DB mixed       : --rw=randrw --rwmixread=75  --bs=8k  --numjobs=8  --iodepth=16
> > >      high-concur.   : --rw=randrw --rwmixread=70  --bs=4k  --numjobs=16 --iodepth=8
> > >      OLTP           : --rw=randrw --rwmixread=70  --bs=16k --numjobs=6  --iodepth=16
> > >      partial-stripe : --rw=randwrite              --bs=8k  --numjobs=4  --iodepth=32
> > >
> > > Each patch stands on its own; I am happy to drop or defer any that is not
> > > justified on its own merit.
> > >
> > > Functional testing on RAID5 and RAID6: create, fail a member, rebuild
> > > onto a spare / re-add, full data read-back verified, and scrub
> > > ("check") reporting mismatch_cnt == 0.  The series was also exercised
> > > with KASAN and lockdep enabled -- including heavy group_thread_cnt
> > > churn on a multi-node setup to stress the per-NUMA-node worker
> > > allocation and the raid5_quiesce hash-lock-all path -- with no reports.
> > >
> > > Hiroshi Nishida (8):
> > >    md: change chunk_sectors and stripe cache counts to unsigned int
> > >    md/raid5: raise stripe cache limit from 32768 to 262144
> > >    md: widen badblock sectors param from int to sector_t
> > >    md/raid5: raise NR_STRIPE_HASH_LOCKS from 8 to 32
> > >    md/raid5: submit a window of stripes during resync/recovery
> > >    md/raid5: allocate worker groups per NUMA node
> > >    md/raid5: raise MAX_STRIPE_BATCH from 8 to 32
> > >    md/raid5: reserve stripe cache for user I/O during rebuild
> > >
> > >   drivers/md/md.c    |   4 +-
> > >   drivers/md/md.h    |  10 ++--
> > >   drivers/md/raid5.c | 129 ++++++++++++++++++++++++++++++++-------------
> > >   drivers/md/raid5.h |  33 ++++++++----
> > >   4 files changed, 121 insertions(+), 55 deletions(-)
> > >
> > > base-commit: 55b77337bdd088c77461588e5ec094421b89911b
> > >
> > --
> > Thanks,
> > Kuai
>
>
>
> --
> Hiroshi Nishida
> nishidafmly@xxxxxxxxx



-- 
Hiroshi Nishida
nishidafmly@xxxxxxxxx