Re: PROBLEM: Kernel BUG with raid5 soft + Xen + DRBD - invalid opcode

[MasterPrenium]

Hello,

Replies below + :
- I don't know if this can help but after the crash, when the system 
reboots, the Raid 5 stack is re-synchronizing
[   37.028239] md10: Warning: Device sdc1 is misaligned
[   37.028541] created bitmap (15 pages) for device md10
[   37.030433] md10: bitmap initialized from disk: read 1 pages, set 59 
of 29807 bits

- Sometimes the kernel completely crash (lost serial + network 
connection), sometimes only got the "BUG" dump, but still have network 
access (but a reboot is impossible, need to reset the system).

- You can find blktrace here (while running fio), I hope it's complete 
since the end of the file is when the kernel crashed : 
https://goo.gl/X9jZ50

Thanks,
MasterPrenium

Le 05/01/2017 à 20:37, Shaohua Li a écrit :
> On Thu, Jan 05, 2017 at 03:16:53PM +0100, MasterPrenium wrote:
> > Hi Shaohua,
> >
> > Thanks for your reply.
> >
> > Let me explain my "huge". For example, if I'm making a low rate i/o stream,
> > I don't get a crash (<1MB written / sec) with random i/o, but if I'm making
> > a random I/O of about 20MB/sec, the kernel crashes in a few minutes (for
> > example, making an rsync, or even synchronising my DRBD stack is causing the
> > crash).
> > I don't know if this can help, but in most of case, when the kernel crashes,
> > after a reboot, my raid 5 stack is re-synchronizing.
> >
> > I'm not able to reproduce the crash with a raw RAID5 stack (with dd/fio
> > ...).
> >
> > It seems I need to stack filesystems to help reproduce it:
> >
> > Here is a configuration test, command lines to explain (the way I'm able to
> > reproduce the crash). Everything is done in dom0.
> > - mdadm --create /dev/md10 --raid-devices=3 --level=5 /dev/sdc1 /dev/sdd1
> > /dev/sde1
> > - mkfs.btrfs /dev/md10
> > - mkdir /tmp/btrfs /mnt/XenVM /tmp/ext4
> > - mount /dev/md10 /tmp/btrfs
> > - btrfs subvolume create /tmp/btrfs/XenVM
> > - umount /tmp/btrfs
> > - mount /dev/md10 /mnt/XenVM -osubvol=XenVM
> > - truncate /mnt/XenVM/VMTestFile.dat -s 800G
> > - mkfs.ext4 /mnt/XenVM/VMTestFile.dat
> > - mount /mnt/XenVM/VMTestFile.dat /tmp/ext4
> >
> > -> Doing this, doesn't seem to crash the kernel :
> > fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite
> > --rwmixwrite=95 --bs=1M --direct=1 --size=80G --numjobs=8 --runtime=600
> > --group_reporting --filename=/mnt/XenVM/Fio.dat
> >
> > -> Doing this, is crashing the kernel in a few minutes :
> > fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite
> > --rwmixwrite=95 --bs=1M --direct=1 --size=80G --numjobs=8 --runtime=600
> > --group_reporting --filename=/tmp/ext4/ext4.dat
> >
> > Note : --direct=1 or --direct=0 doesn't seem to change the behaviour. Also
> > having the raid 5 stack re-synchronizing or already synchronized, doesn't
> > change the behaviour.
> >
> > Here another "crash" : http://pastebin.com/uqLzL4fn
> I'm trying to reproduce, but no success. So
> ext4->btrfs->raid5, crash
> btrfs->raid5, no crash
> right? does subvolume matter? When you create the raid5 array, does adding
> '--assume-clean' option change the behavior? I'd like to narrow down the issue.
> If you can capture the blktrace to the raid5 array, it would be great to hint
> us what kind of IO it is.
>   
Yes Correct.
The subvolume doesn't matter.
-- assume-clean doesn't change the behaviour.
Don't forget that the system needs to be running on xen to crash, 
without (on native kernel) it doesn't crash (or at least, I was not able 
to make it crash).
> > Regarding your patch, I can't find it. Is it the one sent by Konstantin
> > Khlebnikov ?
> Right.
It doesn't help :(. Maybe the crash is happening a little bit later.
> > Do you want the "ext4.dat" fio file ? It will be really difficult for me to
> > provide it to you as I've only a poor ADSL network connection.
> Not necessary.
>
> Thanks,
> Shaohua
>
> > Thanks for your help,
> >
> > MasterPrenium
> >
> > Le 04/01/2017 à 23:30, Shaohua Li a écrit :
> > > On Fri, Dec 23, 2016 at 07:25:56PM +0100, MasterPrenium wrote:
> > > > Hello Guys,
> > > >
> > > > I've having some trouble on a new system I'm setting up. I'm getting a kernel BUG message, seems to be related with the use of Xen (when I boot the system _without_ Xen, I don't get any crash).
> > > > Here is configuration :
> > > > - 3x Hard Drives running on RAID 5 Software raid created by mdadm
> > > > - On top of it, DRBD for replication over another node (Active/passive cluster)
> > > > - On top of it, a BTRFS FileSystem with a few subvolumes
> > > > - On top of it, XEN VMs running.
> > > >
> > > > The BUG is happening when I'm making "huge" I/O (20MB/s with a rsync for example) on the RAID5 stack.
> > > > I've to reset system to make it work again.
> > > what did you mean 'huge' I/O (20M/s)? Is it possible you can reproduce the
> > > issue with a raw raid5 raid? It would be even better if you can give me a fio
> > > job file with the issue, so I can easily debug it.
> > >
> > > also please check if upstream patch (e8d7c33 md/raid5: limit request size
> > > according to implementation limits) helps.
> > >
> > > Thanks,
> > > Shaohua