Re: Filesystem corruption when adding a new RAID device (delayed-resync, write-mostly)

[Mateusz Jończyk]

Dnia 24 lipca 2024 23:19:06 CEST, Paul E Luse <paul.e.luse@xxxxxxxxxxxxxxx> napisał/a:
>On Wed, 24 Jul 2024 22:35:49 +0200
>Mateusz Jończyk <mat.jonczyk@xxxxx> wrote:
>
>> W dniu 22.07.2024 o 07:39, Mateusz Jończyk pisze:
>> > W dniu 20.07.2024 o 16:47, Mateusz Jończyk pisze:
>> >> Hello,
>> >>
>> >> In my laptop, I used to have two RAID1 arrays on top of NVMe and
>> >> SATA SSD drives: /dev/md0 for /boot (not partitioned), /dev/md1
>> >> for remaining data (LUKS
>> >> + LVM + ext4). For performance, I have marked the RAID component
>> >> device for /dev/md1 on the SATA SSD drive write-mostly, which
>> >> "means that the 'md' driver will avoid reading from these devices
>> >> if at all possible" (man mdadm).
>> >>
>> >> Recently, the NVMe drive started having problems (PCI AER errors
>> >> and the controller disappearing), so I removed it from the arrays
>> >> and wiped it. However, I have reseated the drive in the M.2 socket
>> >> and this apparently fixed it (verified with tests).
>> >>
>> >>     $ cat /proc/mdstat
>> >>     Personalities : [raid1] [linear] [multipath] [raid0] [raid6]
>> >> [raid5] [raid4] [raid10] md1 : active raid1 sdb5[1](W)
>> >>           471727104 blocks super 1.2 [2/1] [_U]
>> >>           bitmap: 4/4 pages [16KB], 65536KB chunk
>> >>
>> >>     md2 : active (auto-read-only) raid1 sdb6[3](W) sda1[2]
>> >>           3142656 blocks super 1.2 [2/2] [UU]
>> >>           bitmap: 0/1 pages [0KB], 65536KB chunk
>> >>
>> >>     md0 : active raid1 sdb4[3]
>> >>           2094080 blocks super 1.2 [2/1] [_U]
>> >>          
>> >>     unused devices: <none>
>> >>
>> >> (md2 was used just for testing, ignore it).
>> >>
>> >> Today, I have tried to add the drive back to the arrays by using a
>> >> script that executed in quick succession:
>> >>
>> >>     mdadm /dev/md0 --add --readwrite /dev/nvme0n1p2
>> >>     mdadm /dev/md1 --add --readwrite /dev/nvme0n1p3
>> >>
>> >> This was on Linux 6.10.0, patched with my previous patch:
>> >>
>> >>     https://lore.kernel.org/linux-raid/20240711202316.10775-1-mat.jonczyk@xxxxx/
>> >>
>> >> (which fixed a regression in the kernel and allows it to start
>> >> /dev/md1 with a single drive in write-mostly mode).
>> >> In the background, I was running "rdiff-backup --compare" that was
>> >> comparing data between my array contents and a backup attached via
>> >> USB.
>> >>
>> >> This, however resulted in mayhem - I was unable to start any
>> >> program with an input-output error, etc. I used SysRQ + C to save
>> >> a kernel log:
>> >>
>> > Hello,
>> >
>> > It is possible that my second SSD has some problems and high read
>> > activity during RAID resync triggered it. Reads from that drive are
>> > now very slow (between 10 - 30 MB/s) and this suggests that
>> > something is not OK.
>> 
>> Hello,
>> 
>> Unfortunately, hardware failure seems not to be the case.
>> 
>> I did test it again on 6.10, twice, and in both cases I got
>> filesystem corruption (but not as severe).
>> 
>> On Linux 6.1.96 it seems to be working well (also did two tries).
>> 
>> Please note: in my tests, I was using a RAID component device with
>> a write-mostly bit set. This setup does not work on 6.9+ out of the
>> box and requires the following patch:
>> 
>> commit 36a5c03f23271 ("md/raid1: set max_sectors during early return
>> from choose_slow_rdev()")
>> 
>> that is in master now.
>> 
>> It is also heading into stable, which I'm going to interrupt.
>
>Hi Mateusz,
>
>I'm pretty interested in what is happening here especially as it
>relates to write-mostly.  Couple of questions for you:
>
>1) Are you able to find a simpler reproduction for this, for example
>without mixing SATA and NVMe.  Maybe just using two known good NVMe
>SSDs and follow your steps to repro?

Hello,

Well, I have three drives in my laptop: NVMe, SATA SSD (in the DVD bay) and SATA HDD (platter). I could do tests on top of these two SATA drives.
But maybe it would be easier for me to bisect (or guess-bisect) in the current setup, I haven't made up my mind yet.

>
>2) I don't fully understand your last two statements, maybe you can
>clarify?  With your max_sectors patch does it pass or fail?  If pass,
>what do mean by "I'm going to interrupt"? It sounds like you mean the
>patch doesn't work and you are trying to stop it??

Without this patch I wouldn't be able to do the tests. Without it, degraded RAID1 with a single drive in write-mostly mode doesn’t start at all.

With my last statement I meant that I was going to stop this patch from going to stable kernels. At this point, it doesn’t seem to me that my patch
is the direct cause of the problems, that I missed something. However, I think that it is currently better to fail this setup outright rather than risk
somebody's data.

I have made further tests:

- vanilla 6.8.0 with a write-mostly drive works correctly,

- vanilla 6.10-rc6 without the write mostly bit set also works correctly. 

So it seems that the problem happens only with the write-mostly mode and after 6.8.0.

Greetings,

Mateusz