Re: Filesystem corruption when adding a new RAID device (delayed-resync, write-mostly)

[Paul E Luse]

On Thu, 25 Jul 2024 09:15:40 +0200
Mateusz Jończyk <mat.jonczyk@xxxxx> wrote:

> Dnia 24 lipca 2024 23:19:06 CEST, Paul E Luse
> <paul.e.luse@xxxxxxxxxxxxxxx> napisał/a:
> >On Wed, 24 Jul 2024 22:35:49 +0200
> >Mateusz Jończyk <mat.jonczyk@xxxxx> wrote:
> >
> >> W dniu 22.07.2024 o 07:39, Mateusz Jończyk pisze:
> >> > W dniu 20.07.2024 o 16:47, Mateusz Jończyk pisze:
> >> >> Hello,
> >> >>
> >> >> In my laptop, I used to have two RAID1 arrays on top of NVMe and
> >> >> SATA SSD drives: /dev/md0 for /boot (not partitioned), /dev/md1
> >> >> for remaining data (LUKS
> >> >> + LVM + ext4). For performance, I have marked the RAID component
> >> >> device for /dev/md1 on the SATA SSD drive write-mostly, which
> >> >> "means that the 'md' driver will avoid reading from these
> >> >> devices if at all possible" (man mdadm).
> >> >>
> >> >> Recently, the NVMe drive started having problems (PCI AER errors
> >> >> and the controller disappearing), so I removed it from the
> >> >> arrays and wiped it. However, I have reseated the drive in the
> >> >> M.2 socket and this apparently fixed it (verified with tests).
> >> >>
> >> >>     $ cat /proc/mdstat
> >> >>     Personalities : [raid1] [linear] [multipath] [raid0] [raid6]
> >> >> [raid5] [raid4] [raid10] md1 : active raid1 sdb5[1](W)
> >> >>           471727104 blocks super 1.2 [2/1] [_U]
> >> >>           bitmap: 4/4 pages [16KB], 65536KB chunk
> >> >>
> >> >>     md2 : active (auto-read-only) raid1 sdb6[3](W) sda1[2]
> >> >>           3142656 blocks super 1.2 [2/2] [UU]
> >> >>           bitmap: 0/1 pages [0KB], 65536KB chunk
> >> >>
> >> >>     md0 : active raid1 sdb4[3]
> >> >>           2094080 blocks super 1.2 [2/1] [_U]
> >> >>          
> >> >>     unused devices: <none>
> >> >>
> >> >> (md2 was used just for testing, ignore it).
> >> >>
> >> >> Today, I have tried to add the drive back to the arrays by
> >> >> using a script that executed in quick succession:
> >> >>
> >> >>     mdadm /dev/md0 --add --readwrite /dev/nvme0n1p2
> >> >>     mdadm /dev/md1 --add --readwrite /dev/nvme0n1p3
> >> >>
> >> >> This was on Linux 6.10.0, patched with my previous patch:
> >> >>
> >> >>     https://lore.kernel.org/linux-raid/20240711202316.10775-1-mat.jonczyk@xxxxx/
> >> >>
> >> >> (which fixed a regression in the kernel and allows it to start
> >> >> /dev/md1 with a single drive in write-mostly mode).
> >> >> In the background, I was running "rdiff-backup --compare" that
> >> >> was comparing data between my array contents and a backup
> >> >> attached via USB.
> >> >>
> >> >> This, however resulted in mayhem - I was unable to start any
> >> >> program with an input-output error, etc. I used SysRQ + C to
> >> >> save a kernel log:
> >> >>
> >> > Hello,
> >> >
> >> > It is possible that my second SSD has some problems and high read
> >> > activity during RAID resync triggered it. Reads from that drive
> >> > are now very slow (between 10 - 30 MB/s) and this suggests that
> >> > something is not OK.
> >> 
> >> Hello,
> >> 
> >> Unfortunately, hardware failure seems not to be the case.
> >> 
> >> I did test it again on 6.10, twice, and in both cases I got
> >> filesystem corruption (but not as severe).
> >> 
> >> On Linux 6.1.96 it seems to be working well (also did two tries).
> >> 
> >> Please note: in my tests, I was using a RAID component device with
> >> a write-mostly bit set. This setup does not work on 6.9+ out of the
> >> box and requires the following patch:
> >> 
> >> commit 36a5c03f23271 ("md/raid1: set max_sectors during early
> >> return from choose_slow_rdev()")
> >> 
> >> that is in master now.
> >> 
> >> It is also heading into stable, which I'm going to interrupt.
> >
> >Hi Mateusz,
> >
> >I'm pretty interested in what is happening here especially as it
> >relates to write-mostly.  Couple of questions for you:
> >
> >1) Are you able to find a simpler reproduction for this, for example
> >without mixing SATA and NVMe.  Maybe just using two known good NVMe
> >SSDs and follow your steps to repro?
> 
> Hello,
> 
> Well, I have three drives in my laptop: NVMe, SATA SSD (in the DVD
> bay) and SATA HDD (platter). I could do tests on top of these two
> SATA drives. But maybe it would be easier for me to bisect (or
> guess-bisect) in the current setup, I haven't made up my mind yet.
> 

OK, thanks.

> >
> >2) I don't fully understand your last two statements, maybe you can
> >clarify?  With your max_sectors patch does it pass or fail?  If pass,
> >what do mean by "I'm going to interrupt"? It sounds like you mean the
> >patch doesn't work and you are trying to stop it??
> 
> Without this patch I wouldn't be able to do the tests. Without it,
> degraded RAID1 with a single drive in write-mostly mode doesn’t start
> at all.
> 
> With my last statement I meant that I was going to stop this patch
> from going to stable kernels. At this point, it doesn’t seem to me
> that my patch is the direct cause of the problems, that I missed
> something. However, I think that it is currently better to fail this
> setup outright rather than risk somebody's data.
> 

OK, I would say please do not try to stop the patch, it is a good fix
although maybe not completely solving your problem it should land.

Unless Kwai has another opinion.

-Paul

> I have made further tests:
> 
> - vanilla 6.8.0 with a write-mostly drive works correctly,
> 
> - vanilla 6.10-rc6 without the write mostly bit set also works
> correctly. 
> 
> So it seems that the problem happens only with the write-mostly mode
> and after 6.8.0.
> 
> Greetings,
> 
> Mateusz
> 
>