Re: [PATCH md-6.13] md: remove bitmap file support

From: Yu Kuai
Date: Mon Nov 11 2024 - 08:04:46 EST

Hi,

在 2024/11/11 19:59, Dragan Milivojević 写道:
On 11/11/2024 03:04, Yu Kuai wrote:

Yes, as I said please show me you how you creat the array and your test
script. I must know what you are testing, like single threaded or high
concurrency. For example your result shows bitmap none close to bitmap
external, this is impossible in our previous results. I can only guess
that you're testing single threaded.

All of that is included in the previously linked pastebin.

TBO, I don't know what is this. :(
I will include the contents of that pastebin at the end of this email if that helps.
Every test includes the mdadm create line, disk settings, md settings, fio test line
used and the results and the typical iostat output during the test. I hope that is
sufficient.

BTW, it'll be great if you can provide some perf results of the internal
bitmap in your case, that will show us directly where is the bottleneck.

Not right now, this server is in production and I'm not sure if I will be able
to get it to an idle state or to find the time to do it due to other projects.

BTW do you guys do performance tests? All of the raid levels are

We do, but we never test external bitmap.

I wasn't referring to that, more to the fact that there is a huge difference in
performance between no bitmap and bitmap or that raid (even "simple" levels like 0)
do not scale with real world workloads.

Yes, this is a known problem, the gap here is that I don't think
external bitmap is much helpful, while your result disagree.


The contents of that pastebin, hopefully my email client won't mess up the formating:


5 disk RAID5, 64K chunk

Summary

Test                   BW         IOPS
bitmap internal 64M    700KiB/s   174
bitmap internal 128M   702KiB/s   175
bitmap internal 512M   1142KiB/s  285
bitmap internal 1024M  40.4MiB/s  10.3k
bitmap internal 2G     66.5MiB/s  17.0k
bitmap external 64M    67.8MiB/s  17.3k
bitmap external 1024M  76.5MiB/s  19.6k
bitmap none            80.6MiB/s  20.6k
Single disk 1K         54.1MiB/s  55.4k
Single disk 4K         269MiB/s   68.8k




AlmaLinux release 9.4 (Seafoam Ocelot)
5.14.0-427.20.1.el9_4


nvme list
Node                  Generic               SN Model                                    Namespace Usage                      Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            1460A0F9TSTJ         Dell DC NVMe CD8 U.2 960GB               0x1        122.33  GB / 960.20  GB 512   B +  0 B   2.0.0
/dev/nvme1n1          /dev/ng1n1            S6WRNJ0WA04045P      Samsung SSD 980 PRO with Heatsink 2TB    0x1          0.00   B /   2.00  TB 512   B +  0 B   5B2QGXA7
/dev/nvme2n1          /dev/ng2n1            S6WRNJ0WA04048B      Samsung SSD 980 PRO with Heatsink 2TB    0x1          0.00   B /   2.00  TB 512   B +  0 B   5B2QGXA7
/dev/nvme3n1          /dev/ng3n1            S6WRNJ0W810396H      Samsung SSD 980 PRO with Heatsink 2TB    0x1          0.00   B /   2.00  TB 512   B +  0 B   5B2QGXA7
/dev/nvme4n1          /dev/ng4n1            S6WRNJ0W808149N      Samsung SSD 980 PRO with Heatsink 2TB    0x1          0.00   B /   2.00  TB 512   B +  0 B   5B2QGXA7
/dev/nvme5n1          /dev/ng5n1            S6WRNJ0WA04043Z      Samsung SSD 980 PRO with Heatsink 2TB    0x1          0.00   B /   2.00  TB 512   B +  0 B   5B2QGXA7
/dev/nvme6n1          /dev/ng6n1            PHBT909504AH016N     INTEL MEMPEK1J016GAL                     0x1         14.40  GB /  14.40  GB 512   B +  0 B   K4110420
/dev/nvme7n1          /dev/ng7n1            S6WRNJ0WA04036R      Samsung SSD 980 PRO with Heatsink 2TB    0x1          0.00   B /   2.00  TB 512   B +  0 B   5B2QGXA7
/dev/nvme8n1          /dev/ng8n1            S6WRNJ0WA04050H      Samsung SSD 980 PRO with Heatsink 2TB    0x1          0.00   B /   2.00  TB 512   B +  0 B   5B2QGXA7




bitmap internal 64M
================================================================
mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=64M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1

for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
blockdev --setra 1024 /dev/md/raid5

echo 8 > /sys/block/md127/md/group_thread_cnt
echo 8192 > /sys/block/md127/md/stripe_cache_size


The array set up is fine. And the following external bitmap is using
/bitmap/bitmap.bin, does the back-end storage of this file the same as
test device?


fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5

Then this is what I suspected, the above test is quite limited and can't
replace the real world workload, 1 thread 1 iodepth with 4k randwrite.

I still can't believe your test result, and I can't figure out why
internal bitmap is so slow. Hence I use ramdisk(10GB) to create a raid5,
and use the same fio script to test, the result is quite different from
yours:

ram0: 981MiB/s
non-bitmap: 132MiB/s
internal-bitmap: 95.5MiB/s

Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=716KiB/s][w=179 IOPS][eta 00m:00s]
Raid5: (groupid=0, jobs=1): err= 0: pid=4718: Sun Jun 30 02:18:30 2024
  write: IOPS=174, BW=700KiB/s (717kB/s)(41.0MiB/60005msec); 0 zone resets
    slat (usec): min=4, max=18062, avg=11.28, stdev=176.21
    clat (usec): min=46, max=13308, avg=5700.08, stdev=1194.59
     lat (usec): min=53, max=22717, avg=5711.36, stdev=1206.03
    clat percentiles (usec):
     |  1.00th=[   51],  5.00th=[ 5800], 10.00th=[ 5800], 20.00th=[ 5866],
     | 30.00th=[ 5866], 40.00th=[ 5866], 50.00th=[ 5866], 60.00th=[ 5932],
     | 70.00th=[ 5932], 80.00th=[ 5932], 90.00th=[ 5932], 95.00th=[ 5997],
     | 99.00th=[ 6194], 99.50th=[ 8586], 99.90th=[10290], 99.95th=[13042],
     | 99.99th=[13042]
   bw (  KiB/s): min=  608, max=  752, per=100.00%, avg=700.03, stdev=20.93, samples=119

There is absolutely something wrong here, it doesn't make sense to me
that internal bitmap is so slow. However, I have no idea until you can
provide the perf result.

Thanks,
Kuai

   iops        : min=  152, max=  188, avg=175.01, stdev= 5.23, samples=119
  lat (usec)   : 50=0.68%, 100=3.23%
  lat (msec)   : 10=95.99%, 20=0.10%
  cpu          : usr=0.08%, sys=0.24%, ctx=10503, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10499,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=700KiB/s (717kB/s), 700KiB/s-700KiB/s (717kB/s-717kB/s), io=41.0MiB (43.0MB), run=60005-60005msec


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.07    0.00    0.00   99.93

Device   r/s    rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s     wMB/s wrqm/s  %wrqm  w_await  wareq-sz  f/s     f_await  aqu-sz  %util
md127    0.00   0.00   0.00    0.00   0.00     0.00      175.47  0.69 0.00    0.00   5.68     4.00      0.00    0.00     1.00    100.00
nvme1n1  69.40  0.27   0.00    0.00   0.01     4.00      237.80  0.93 0.00    0.00   3.55     4.00      168.47  0.81     0.98    95.59
nvme2n1  69.20  0.27   0.00    0.00   0.01     4.00      237.60  0.93 0.00    0.00   3.55     4.00      168.47  0.81     0.98    95.61
nvme3n1  72.20  0.28   0.00    0.00   0.01     4.00      240.60  0.94 0.00    0.00   3.51     4.00      168.47  0.83     0.98    95.29
nvme4n1  68.07  0.27   0.00    0.00   0.02     4.00      236.53  0.92 0.00    0.00   3.57     4.00      168.47  0.81     0.98    95.65
nvme5n1  72.07  0.28   0.00    0.00   0.02     4.00      240.53  0.94 0.00    0.00   3.52     4.00      168.47  0.83     0.99    95.31


mdadm -X /dev/nvme1n1
        Filename : /dev/nvme1n1
           Magic : 6d746962
         Version : 4
            UUID : 77fa1a1b:2f0dd646:adc85c8e:985513a8
          Events : 3
  Events Cleared : 3
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
          Bitmap : 29807 bits (chunks), 1517 dirty (5.1%)


bitmap internal 128M
================================================================
for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done

mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=128M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1

for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
blockdev --setra 1024 /dev/md/raid5

echo 8 > /sys/block/md127/md/group_thread_cnt
echo 8192 > /sys/block/md127/md/stripe_cache_size


fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5

Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=716KiB/s][w=179 IOPS][eta 00m:00s]
Raid5: (groupid=0, jobs=1): err= 0: pid=6283: Sun Jun 30 02:49:06 2024
  write: IOPS=175, BW=702KiB/s (719kB/s)(41.1MiB/60002msec); 0 zone resets
    slat (usec): min=8, max=18200, avg=16.06, stdev=177.21
    clat (usec): min=61, max=20048, avg=5675.78, stdev=1968.88
     lat (usec): min=74, max=22975, avg=5691.84, stdev=1976.14
    clat percentiles (usec):
     |  1.00th=[   68],  5.00th=[   73], 10.00th=[ 5866], 20.00th=[ 5932],
     | 30.00th=[ 5932], 40.00th=[ 5932], 50.00th=[ 5932], 60.00th=[ 5997],
     | 70.00th=[ 5997], 80.00th=[ 5997], 90.00th=[ 5997], 95.00th=[ 6063],
     | 99.00th=[14615], 99.50th=[15008], 99.90th=[16188], 99.95th=[16319],
     | 99.99th=[16319]
   bw (  KiB/s): min=  384, max=  816, per=99.97%, avg=702.12, stdev=72.52, samples=119
   iops        : min=   96, max=  204, avg=175.53, stdev=18.13, samples=119
  lat (usec)   : 100=7.62%, 250=0.01%
  lat (msec)   : 10=90.80%, 20=1.56%, 50=0.01%
  cpu          : usr=0.11%, sys=0.34%, ctx=10539, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10534,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=702KiB/s (719kB/s), 702KiB/s-702KiB/s (719kB/s-719kB/s), io=41.1MiB (43.1MB), run=60002-60002msec





avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.08    0.00    0.00   99.92

Device   r/s    rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s     wMB/s wrqm/s  %wrqm  w_await  wareq-sz  f/s     f_await  aqu-sz  %util
md127    0.00   0.00   0.00    0.00   0.00     0.00      173.73  0.68 0.00    0.00   5.73     4.00      0.00    0.00     1.00    99.99
nvme1n1  65.87  0.26   0.00    0.00   0.01     4.00      226.07  0.65 0.00    0.00   3.60     2.94      160.20  0.81     0.94    92.46
nvme2n1  71.33  0.28   0.00    0.00   0.02     4.00      231.53  0.67 0.00    0.00   3.50     2.96      160.27  0.84     0.95    91.79
nvme3n1  68.60  0.27   0.00    0.00   0.02     4.00      228.80  0.66 0.00    0.00   3.68     2.95      160.27  0.93     0.99    94.37
nvme4n1  68.87  0.27   0.00    0.00   0.02     4.00      229.07  0.66 0.00    0.00   3.52     2.95      160.20  0.81     0.94    91.59
nvme5n1  72.80  0.28   0.00    0.00   0.02     4.00      233.00  0.68 0.00    0.00   3.53     2.97      160.27  0.87     0.96    92.29


mdadm -X /dev/nvme1n1
        Filename : /dev/nvme1n1
           Magic : 6d746962
         Version : 4
            UUID : 93fdcd4b:ae61a1f8:4d809242:2cd4a4c7
          Events : 3
  Events Cleared : 3
           State : OK
       Chunksize : 128 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
          Bitmap : 14904 bits (chunks), 1617 dirty (10.8%)





bitmap internal 512M
================================================================
for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done

mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=512M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1

for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
blockdev --setra 1024 /dev/md/raid5

echo 8 > /sys/block/md127/md/group_thread_cnt
echo 8192 > /sys/block/md127/md/stripe_cache_size


fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5


Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=1232KiB/s][w=308 IOPS][eta 00m:00s]
Raid5: (groupid=0, jobs=1): err= 0: pid=6661: Sun Jun 30 02:58:11 2024
  write: IOPS=285, BW=1142KiB/s (1169kB/s)(66.9MiB/60006msec); 0 zone resets
    slat (usec): min=4, max=18130, avg=10.80, stdev=138.54
    clat (usec): min=42, max=13261, avg=3490.08, stdev=2945.95
     lat (usec): min=50, max=22827, avg=3500.88, stdev=2949.63
    clat percentiles (usec):
     |  1.00th=[   49],  5.00th=[   51], 10.00th=[   52], 20.00th=[   55],
     | 30.00th=[   58], 40.00th=[   72], 50.00th=[ 5866], 60.00th=[ 5932],
     | 70.00th=[ 5932], 80.00th=[ 5932], 90.00th=[ 5997], 95.00th=[ 5997],
     | 99.00th=[ 6128], 99.50th=[ 8586], 99.90th=[ 9896], 99.95th=[13042],
     | 99.99th=[13042]
   bw (  KiB/s): min=  600, max= 1648, per=99.68%, avg=1138.89, stdev=188.44, samples=119
   iops        : min=  150, max=  412, avg=284.72, stdev=47.11, samples=119
  lat (usec)   : 50=3.41%, 100=38.62%, 250=0.04%, 500=0.03%
  lat (msec)   : 10=57.83%, 20=0.07%
  cpu          : usr=0.09%, sys=0.40%, ctx=17130, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,17127,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=1142KiB/s (1169kB/s), 1142KiB/s-1142KiB/s (1169kB/s-1169kB/s), io=66.9MiB (70.2MB), run=60006-60006msec


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.10    0.00    0.00   99.90

Device   r/s     rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s     wMB/s wrqm/s  %wrqm  w_await  wareq-sz  f/s     f_await  aqu-sz  %util
md127    0.00    0.00   0.00    0.00   0.00     0.00      307.13  1.20 0.00    0.00   3.24     4.00      0.00    0.00     1.00    100.00
nvme1n1  120.47  0.47   0.00    0.00   0.01     4.00      286.07  0.63 0.00    0.00   3.03     2.26      165.60  0.99     1.03    96.58
nvme2n1  123.87  0.48   0.00    0.00   0.01     4.00      289.47  0.65 0.00    0.00   3.00     2.28      165.60  1.00     1.04    96.63
nvme3n1  120.87  0.47   0.00    0.00   0.01     4.00      286.47  0.63 0.00    0.00   3.02     2.27      165.60  1.00     1.03    96.39
nvme4n1  125.00  0.49   0.00    0.00   0.02     4.00      290.60  0.65 0.00    0.00   3.00     2.29      165.60  1.02     1.04    96.54
nvme5n1  124.07  0.48   0.00    0.00   0.02     4.00      289.67  0.65 0.00    0.00   3.01     2.28      165.60  1.03     1.04    96.59


mdadm -X /dev/nvme1n1
        Filename : /dev/nvme1n1
           Magic : 6d746962
         Version : 4
            UUID : 17eadc76:a367542a:feb6e24e:d650576c
          Events : 3
  Events Cleared : 3
           State : OK
       Chunksize : 512 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
          Bitmap : 3726 bits (chunks), 1977 dirty (53.1%)


bitmap internal 1024M
================================================================
for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done

mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=1024M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1

for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
blockdev --setra 1024 /dev/md/raid5

echo 8 > /sys/block/md127/md/group_thread_cnt
echo 8192 > /sys/block/md127/md/stripe_cache_size


fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5

fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=51.0MiB/s][w=13.1k IOPS][eta 00m:00s]
Raid5: (groupid=0, jobs=1): err= 0: pid=7120: Sun Jun 30 03:08:12 2024
  write: IOPS=10.3k, BW=40.4MiB/s (42.4MB/s)(2425MiB/60001msec); 0 zone resets
    slat (usec): min=6, max=18135, avg= 8.93, stdev=23.41
    clat (usec): min=3, max=10459, avg=86.97, stdev=342.95
     lat (usec): min=63, max=22927, avg=95.90, stdev=344.33
    clat percentiles (usec):
     |  1.00th=[   62],  5.00th=[   63], 10.00th=[   64], 20.00th=[   65],
     | 30.00th=[   65], 40.00th=[   66], 50.00th=[   67], 60.00th=[   67],
     | 70.00th=[   68], 80.00th=[   69], 90.00th=[   70], 95.00th=[   74],
     | 99.00th=[  133], 99.50th=[  155], 99.90th=[ 5997], 99.95th=[ 5997],
     | 99.99th=[ 6063]
   bw (  KiB/s): min=  616, max=52968, per=99.80%, avg=41305.95, stdev=20465.79, samples=119
   iops        : min=  154, max=13242, avg=10326.47, stdev=5116.44, samples=119
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 100=98.64%, 250=1.00%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.33%, 20=0.01%
  cpu          : usr=1.89%, sys=12.74%, ctx=620837, majf=0, minf=170751
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,620822,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=40.4MiB/s (42.4MB/s), 40.4MiB/s-40.4MiB/s (42.4MB/s-42.4MB/s), io=2425MiB (2543MB), run=60001-60001msec


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.04    0.00    1.27    0.00    0.00   98.70

Device   r/s      rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s wMB/s  wrqm/s  %wrqm  w_await  wareq-sz  f/s   f_await  aqu-sz  %util
md127    0.00     0.00   0.00    0.00   0.00     0.00      18216.93 71.16  0.00    0.00   0.05     4.00      0.00  0.00     0.88    100.00
nvme1n1  7256.20  28.34  0.00    0.00   0.01     4.00      7256.27 28.34  0.00    0.00   0.01     4.00      0.00  0.00     0.19    99.71
nvme2n1  7302.53  28.53  0.00    0.00   0.01     4.00      7302.53 28.53  0.00    0.00   0.01     4.00      0.00  0.00     0.17    99.73
nvme3n1  7278.47  28.43  0.00    0.00   0.01     4.00      7278.53 28.43  0.00    0.00   0.01     4.00      0.00  0.00     0.18    99.57
nvme4n1  7303.93  28.53  0.00    0.00   0.01     4.00      7303.93 28.53  0.00    0.00   0.01     4.00      0.00  0.00     0.21    99.74
nvme5n1  7292.67  28.49  0.00    0.00   0.02     4.00      7292.60 28.49  0.00    0.00   0.02     4.00      0.00  0.00     0.22    99.69



mdadm -X /dev/nvme1n1
        Filename : /dev/nvme1n1
           Magic : 6d746962
         Version : 4
            UUID : a0c7ad14:50689e41:e065a166:4935a186
          Events : 3
  Events Cleared : 3
           State : OK
       Chunksize : 1 GB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
          Bitmap : 1863 bits (chunks), 1863 dirty (100.0%)


bitmap internal 2G
================================================================
for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done

mdadm --verbose --create --assume-clean --bitmap=internal --bitmap-chunk=2G /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1

for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
blockdev --setra 1024 /dev/md/raid5

echo 8 > /sys/block/md127/md/group_thread_cnt
echo 8192 > /sys/block/md127/md/stripe_cache_size


fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5

Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=74.7MiB/s][w=19.1k IOPS][eta 00m:00s]
Raid5: (groupid=0, jobs=1): err= 0: pid=7696: Sun Jun 30 03:30:40 2024
  write: IOPS=17.0k, BW=66.5MiB/s (69.8MB/s)(3993MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=18094, avg= 4.79, stdev=17.94
    clat (usec): min=5, max=10352, avg=53.37, stdev=181.29
     lat (usec): min=41, max=22883, avg=58.16, stdev=182.72
    clat percentiles (usec):
     |  1.00th=[   43],  5.00th=[   44], 10.00th=[   45], 20.00th=[   46],
     | 30.00th=[   46], 40.00th=[   47], 50.00th=[   47], 60.00th=[   48],
     | 70.00th=[   48], 80.00th=[   49], 90.00th=[   50], 95.00th=[   52],
     | 99.00th=[   90], 99.50th=[  126], 99.90th=[  873], 99.95th=[ 5997],
     | 99.99th=[ 6063]
   bw (  KiB/s): min=  640, max=80168, per=99.91%, avg=68080.94, stdev=21547.29, samples=119
   iops        : min=  160, max=20042, avg=17020.24, stdev=5386.82, samples=119
  lat (usec)   : 10=0.01%, 50=92.06%, 100=7.10%, 250=0.73%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.09%, 20=0.01%
  cpu          : usr=2.22%, sys=11.55%, ctx=1022167, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1022154,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=66.5MiB/s (69.8MB/s), 66.5MiB/s-66.5MiB/s (69.8MB/s-69.8MB/s), io=3993MiB (4187MB), run=60001-60001msec


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.04    0.00    1.15    0.00    0.00   98.81

Device   r/s      rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s wMB/s  wrqm/s  %wrqm  w_await  wareq-sz  f/s   f_await  aqu-sz  %util
md127    0.00     0.00   0.00    0.00   0.00     0.00      18836.40 73.58  0.00    0.00   0.05     4.00      0.00  0.00     0.87    100.00
nvme1n1  7505.27  29.32  0.00    0.00   0.01     4.00      7505.40 29.32  0.00    0.00   0.01     4.00      0.00  0.00     0.19    99.93
nvme2n1  7510.00  29.34  0.00    0.00   0.01     4.00      7510.07 29.34  0.00    0.00   0.01     4.00      0.00  0.00     0.17    99.90
nvme3n1  7561.40  29.54  0.00    0.00   0.01     4.00      7561.47 29.54  0.00    0.00   0.01     4.00      0.00  0.00     0.19    100.00
nvme4n1  7543.07  29.47  0.00    0.00   0.01     4.00      7543.07 29.47  0.00    0.00   0.01     4.00      0.00  0.00     0.21    99.91
nvme5n1  7552.73  29.50  0.00    0.00   0.01     4.00      7552.80 29.50  0.00    0.00   0.01     4.00      0.00  0.00     0.22    99.91



mdadm -X /dev/nvme1n1
        Filename : /dev/nvme1n1
           Magic : 6d746962
         Version : 4
            UUID : 7d8ed7e8:4c2c4b17:723a22e5:4e9b5200
          Events : 3
  Events Cleared : 3
           State : OK
       Chunksize : 2 GB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
          Bitmap : 932 bits (chunks), 932 dirty (100.0%)









bitmap external 64M
================================================================
for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done

mdadm --verbose --create --assume-clean --bitmap=/bitmap/bitmap.bin --bitmap-chunk=64M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1

for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
blockdev --setra 1024 /dev/md/raid5

echo 8 > /sys/block/md127/md/group_thread_cnt
echo 8192 > /sys/block/md127/md/stripe_cache_size


fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5

Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=67.3MiB/s][w=17.2k IOPS][eta 00m:00s]
Raid5: (groupid=0, jobs=1): err= 0: pid=7912: Sun Jun 30 03:39:11 2024
  write: IOPS=17.3k, BW=67.8MiB/s (71.1MB/s)(4066MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=21987, avg= 6.11, stdev=22.04
    clat (usec): min=3, max=8410, avg=50.79, stdev=27.03
     lat (usec): min=42, max=22140, avg=56.90, stdev=35.13
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   44], 20.00th=[   46],
     | 30.00th=[   47], 40.00th=[   48], 50.00th=[   49], 60.00th=[   50],
     | 70.00th=[   51], 80.00th=[   52], 90.00th=[   56], 95.00th=[   68],
     | 99.00th=[   93], 99.50th=[  124], 99.90th=[  155], 99.95th=[  237],
     | 99.99th=[ 1037]
   bw (  KiB/s): min=38120, max=82576, per=100.00%, avg=69402.96, stdev=7769.33, samples=119
   iops        : min= 9530, max=20644, avg=17350.76, stdev=1942.33, samples=119
  lat (usec)   : 4=0.01%, 20=0.01%, 50=67.87%, 100=31.34%, 250=0.76%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=2.23%, sys=14.27%, ctx=1040947, majf=0, minf=233929
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1040925,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=67.8MiB/s (71.1MB/s), 67.8MiB/s-67.8MiB/s (71.1MB/s-71.1MB/s), io=4066MiB (4264MB), run=60001-60001msec


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.04    0.00    1.15    0.00    0.00   98.81

Device   r/s      rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s wMB/s  wrqm/s  %wrqm  w_await  wareq-sz  f/s   f_await  aqu-sz  %util
md127    0.00     0.00   0.00    0.00   0.00     0.00      18428.60 71.99  0.00    0.00   0.05     4.00      0.00  0.00     0.87    99.99
nvme1n1  7399.40  28.90  0.00    0.00   0.01     4.00      7399.47 28.90  0.00    0.00   0.01     4.00      0.00  0.00     0.17    99.73
nvme2n1  7361.20  28.75  0.00    0.00   0.01     4.00      7361.27 28.75  0.00    0.00   0.01     4.00      0.00  0.00     0.20    99.63
nvme3n1  7376.67  28.82  0.00    0.00   0.01     4.00      7376.73 28.82  0.00    0.00   0.01     4.00      0.00  0.00     0.21    99.63
nvme4n1  7367.27  28.78  0.00    0.00   0.01     4.00      7367.20 28.78  0.00    0.00   0.01     4.00      0.00  0.00     0.18    99.65
nvme5n1  7352.47  28.72  0.00    0.00   0.01     4.00      7352.67 28.72  0.00    0.00   0.01     4.00      0.00  0.00     0.20    99.73
nvme8n1  0.47     0.00   0.00    0.00   0.00     4.00      293.40 1.15   0.00    0.00   0.02     4.00      0.00  0.00     0.01    24.24



mdadm -X /bitmap/bitmap.bin
        Filename : /bitmap/bitmap.bin
           Magic : 6d746962
         Version : 4
            UUID : 1e3480e5:1f9d8b8a:53ebc6b7:279afb73
          Events : 3
  Events Cleared : 3
           State : OK
       Chunksize : 64 MB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
          Bitmap : 29807 bits (chunks), 29665 dirty (99.5%)



bitmap external 1024M
================================================================
for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done

mdadm --verbose --create --assume-clean --bitmap=/bitmap/bitmap.bin --bitmap-chunk=1024M /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1

for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
blockdev --setra 1024 /dev/md/raid5

echo 8 > /sys/block/md127/md/group_thread_cnt
echo 8192 > /sys/block/md127/md/stripe_cache_size


fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5

Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=70.6MiB/s][w=18.1k IOPS][eta 00m:00s]
Raid5: (groupid=0, jobs=1): err= 0: pid=8592: Sun Jun 30 03:54:11 2024
  write: IOPS=19.6k, BW=76.5MiB/s (80.2MB/s)(4590MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=21819, avg= 4.12, stdev=20.16
    clat (usec): min=22, max=3706, avg=46.37, stdev=20.38
     lat (usec): min=40, max=21951, avg=50.49, stdev=28.81
    clat percentiles (usec):
     |  1.00th=[   40],  5.00th=[   41], 10.00th=[   42], 20.00th=[   42],
     | 30.00th=[   43], 40.00th=[   44], 50.00th=[   45], 60.00th=[   47],
     | 70.00th=[   48], 80.00th=[   49], 90.00th=[   50], 95.00th=[   52],
     | 99.00th=[   86], 99.50th=[  120], 99.90th=[  157], 99.95th=[  233],
     | 99.99th=[  906]
   bw (  KiB/s): min=61616, max=84728, per=100.00%, avg=78398.66, stdev=5410.81, samples=119
   iops        : min=15404, max=21182, avg=19599.66, stdev=1352.70, samples=119
  lat (usec)   : 50=90.10%, 100=9.16%, 250=0.72%, 500=0.01%, 750=0.01%
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=2.35%, sys=11.88%, ctx=1175104, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1175086,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=76.5MiB/s (80.2MB/s), 76.5MiB/s-76.5MiB/s (80.2MB/s-80.2MB/s), io=4590MiB (4813MB), run=60001-60001msec


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.04    0.00    1.03    0.00    0.00   98.93

Device   r/s      rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s wMB/s  wrqm/s  %wrqm  w_await  wareq-sz  f/s   f_await  aqu-sz  %util
md127    0.00     0.00   0.00    0.00   0.00     0.00      20758.20 81.09  0.00    0.00   0.04     4.00      0.00  0.00     0.89    100.00
nvme1n1  8291.67  32.39  0.00    0.00   0.01     4.00      8291.73 32.39  0.00    0.00   0.01     4.00      0.00  0.00     0.22    99.87
nvme2n1  8270.93  32.31  0.00    0.00   0.01     4.00      8271.07 32.31  0.00    0.00   0.01     4.00      0.00  0.00     0.19    99.79
nvme3n1  8310.67  32.46  0.00    0.00   0.01     4.00      8310.80 32.46  0.00    0.00   0.01     4.00      0.00  0.00     0.20    99.83
nvme4n1  8300.67  32.42  0.00    0.00   0.01     4.00      8300.67 32.42  0.00    0.00   0.01     4.00      0.00  0.00     0.23    99.76
nvme5n1  8342.13  32.59  0.00    0.00   0.02     4.00      8342.13 32.59  0.00    0.00   0.01     4.00      0.00  0.00     0.25    99.85
nvme8n1  0.33     0.00   0.00    0.00   8.40     4.00      0.00 0.00   0.00    0.00   0.00     0.00      0.00  0.00     0.00    0.33


mdadm -X /bitmap/bitmap.bin
        Filename : /bitmap/bitmap.bin
           Magic : 6d746962
         Version : 4
            UUID : 30e6211d:31ac1204:e6cdadb3:9691d3ee
          Events : 3
  Events Cleared : 3
           State : OK
       Chunksize : 1 GB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 1953382464 (1862.89 GiB 2000.26 GB)
          Bitmap : 1863 bits (chunks), 1863 dirty (100.0%)



bitmap none
================================================================
for dev in /dev/nvme{1..5}n1; do nvme format --force $dev; done

mdadm --verbose --create --assume-clean --bitmap=none /dev/md/raid5 --name=raid5 --level=5 --chunk=64K --raid-devices=5 /dev/nvme{1..5}n1

for dev in /dev/nvme{1..5}n1; do blockdev --setra 256 $dev; done
blockdev --setra 1024 /dev/md/raid5

echo 8 > /sys/block/md127/md/group_thread_cnt
echo 8192 > /sys/block/md127/md/stripe_cache_size


fio --filename=/dev/md/raid5 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Raid5

Raid5: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=82.5MiB/s][w=21.1k IOPS][eta 00m:00s]
Raid5: (groupid=0, jobs=1): err= 0: pid=9158: Sun Jun 30 04:11:01 2024
  write: IOPS=20.6k, BW=80.6MiB/s (84.5MB/s)(4833MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=13598, avg= 3.50, stdev=12.46
    clat (usec): min=4, max=3694, avg=44.31, stdev=21.60
     lat (usec): min=39, max=13681, avg=47.81, stdev=24.98
    clat percentiles (usec):
     |  1.00th=[   39],  5.00th=[   40], 10.00th=[   41], 20.00th=[   41],
     | 30.00th=[   42], 40.00th=[   43], 50.00th=[   43], 60.00th=[   44],
     | 70.00th=[   45], 80.00th=[   46], 90.00th=[   48], 95.00th=[   50],
     | 99.00th=[   87], 99.50th=[  117], 99.90th=[  157], 99.95th=[  229],
     | 99.99th=[  963]
   bw (  KiB/s): min=74112, max=86712, per=100.00%, avg=82486.43, stdev=3696.94, samples=119
   iops        : min=18528, max=21678, avg=20621.59, stdev=924.23, samples=119
  lat (usec)   : 10=0.01%, 50=95.91%, 100=3.33%, 250=0.74%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=2.30%, sys=10.74%, ctx=1237375, majf=0, minf=179597
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1237359,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=80.6MiB/s (84.5MB/s), 80.6MiB/s-80.6MiB/s (84.5MB/s-84.5MB/s), io=4833MiB (5068MB), run=60001-60001msec




avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.04    0.00    1.06    0.00    0.00   98.91

Device   r/s      rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s wMB/s  wrqm/s  %wrqm  w_await  wareq-sz  f/s   f_await  aqu-sz  %util
md127    0.00     0.00   0.00    0.00   0.00     0.00      20040.87 78.28  0.00    0.00   0.04     4.00      0.00  0.00     0.89    99.99
nvme1n1  8016.80  31.32  0.00    0.00   0.01     4.00      8016.93 31.32  0.00    0.00   0.01     4.00      0.00  0.00     0.21    99.68
nvme2n1  7983.20  31.18  0.00    0.00   0.01     4.00      7983.20 31.18  0.00    0.00   0.01     4.00      0.00  0.00     0.18    99.74
nvme3n1  8030.07  31.37  0.00    0.00   0.01     4.00      8030.20 31.37  0.00    0.00   0.01     4.00      0.00  0.00     0.20    99.62
nvme4n1  8016.40  31.31  0.00    0.00   0.01     4.00      8016.40 31.31  0.00    0.00   0.01     4.00      0.00  0.00     0.23    99.73
nvme5n1  8034.87  31.39  0.00    0.00   0.02     4.00      8035.00 31.39  0.00    0.00   0.01     4.00      0.00  0.00     0.24    99.71




single disk 1K RW
================================================================
fio --filename=/dev/nvme8n1 --direct=1 --rw=randwrite --bs=1k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Single
Single: (g=0): rw=randwrite, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=54.3MiB/s][w=55.6k IOPS][eta 00m:00s]
Single: (groupid=0, jobs=1): err= 0: pid=4471: Sun Jun 30 18:31:56 2024
  write: IOPS=55.4k, BW=54.1MiB/s (56.7MB/s)(3244MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=2792, avg= 2.71, stdev= 2.12
    clat (nsec): min=651, max=8350.9k, avg=14864.41, stdev=5360.57
     lat (usec): min=15, max=8403, avg=17.57, stdev= 5.79
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   15], 80.00th=[   16], 90.00th=[   16], 95.00th=[   16],
     | 99.00th=[   18], 99.50th=[   22], 99.90th=[   32], 99.95th=[   33],
     | 99.99th=[  206]
   bw (  KiB/s): min=51884, max=56778, per=100.00%, avg=55394.37, stdev=561.60, samples=119
   iops        : min=51884, max=56778, avg=55394.44, stdev=561.62, samples=119
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=99.43%, 50=0.54%
  lat (usec)   : 100=0.01%, 250=0.02%, 500=0.01%
  lat (msec)   : 10=0.01%
  cpu          : usr=3.57%, sys=16.41%, ctx=3321571, majf=0, minf=180742
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3321653,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=54.1MiB/s (56.7MB/s), 54.1MiB/s-54.1MiB/s (56.7MB/s-56.7MB/s), io=3244MiB (3401MB), run=60001-60001msec

Disk stats (read/write):
  nvme8n1: ios=0/3309968, merge=0/0, ticks=0/44637, in_queue=44638, util=99.71%


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.04    0.00    0.42    0.00    0.00   99.54

Device   r/s   rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s       wMB/s wrqm/s  %wrqm  w_await  wareq-sz  f/s   f_await  aqu-sz  %util
nvme8n1  0.00  0.00   0.00    0.00   0.00     0.00      55496.93  54.20 0.00    0.00   0.01     1.00      0.00  0.00     0.75    100.00





single disk 4K RW
================================================================
blockdev --setra 256 /dev/nvme8n1

fio --filename=/dev/nvme8n1 --direct=1 --rw=randwrite --bs=4k --ioengine=libaio --iodepth=1 --runtime=60 --numjobs=1 --group_reporting --time_based --name=Single

Single: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=270MiB/s][w=69.0k IOPS][eta 00m:00s]
Single: (groupid=0, jobs=1): err= 0: pid=4396: Sun Jun 30 18:21:52 2024
  write: IOPS=68.8k, BW=269MiB/s (282MB/s)(15.8GiB/60001msec); 0 zone resets
    slat (usec): min=2, max=796, avg= 2.45, stdev= 1.59
    clat (nsec): min=652, max=8343.1k, avg=11616.73, stdev=5088.99
     lat (usec): min=11, max=8410, avg=14.06, stdev= 5.36
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   12], 10.00th=[   12], 20.00th=[   12],
     | 30.00th=[   12], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   12], 95.00th=[   12],
     | 99.00th=[   14], 99.50th=[   17], 99.90th=[   28], 99.95th=[   34],
     | 99.99th=[  204]
   bw (  KiB/s): min=264072, max=277568, per=100.00%, avg=275629.71, stdev=1902.45, samples=119
   iops        : min=66018, max=69392, avg=68907.43, stdev=475.55, samples=119
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.04%, 20=99.55%, 50=0.38%
  lat (usec)   : 100=0.01%, 250=0.02%, 1000=0.01%
  lat (msec)   : 10=0.01%
  cpu          : usr=5.20%, sys=21.28%, ctx=4129887, majf=0, minf=45204
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4130258,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=269MiB/s (282MB/s), 269MiB/s-269MiB/s (282MB/s-282MB/s), io=15.8GiB (16.9GB), run=60001-60001msec

Disk stats (read/write):
  nvme8n1: ios=0/4119593, merge=0/0, ticks=0/40922, in_queue=40923, util=99.89%


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.08    0.00    0.57    0.00    0.00   99.35

Device   r/s   rMB/s  rrqm/s  %rrqm  r_await  rareq-sz  w/s wMB/s   wrqm/s  %wrqm  w_await  wareq-sz  f/s   f_await  aqu-sz  %util
nvme8n1  0.00  0.00   0.00    0.00   0.00     0.00      69041.33 269.69  0.00    0.00   0.01     4.00      0.00  0.00     0.68    100.00

.