Re: PROBLEM: Kernel BUG with raid5 soft + Xen + DRBD - invalid opcode

From: MasterPrenium
Date: Sun Jan 08 2017 - 08:32:47 EST


Hello,

Replies below + :
- I don't know if this can help but after the crash, when the system reboots, the Raid 5 stack is re-synchronizing
[ 37.028239] md10: Warning: Device sdc1 is misaligned
[ 37.028541] created bitmap (15 pages) for device md10
[ 37.030433] md10: bitmap initialized from disk: read 1 pages, set 59 of 29807 bits

- Sometimes the kernel completely crash (lost serial + network connection), sometimes only got the "BUG" dump, but still have network access (but a reboot is impossible, need to reset the system).

- You can find blktrace here (while running fio), I hope it's complete since the end of the file is when the kernel crashed : https://goo.gl/X9jZ50

Thanks,
MasterPrenium

Le 05/01/2017 à 20:37, Shaohua Li a écrit :
On Thu, Jan 05, 2017 at 03:16:53PM +0100, MasterPrenium wrote:
Hi Shaohua,

Thanks for your reply.

Let me explain my "huge". For example, if I'm making a low rate i/o stream,
I don't get a crash (<1MB written / sec) with random i/o, but if I'm making
a random I/O of about 20MB/sec, the kernel crashes in a few minutes (for
example, making an rsync, or even synchronising my DRBD stack is causing the
crash).
I don't know if this can help, but in most of case, when the kernel crashes,
after a reboot, my raid 5 stack is re-synchronizing.

I'm not able to reproduce the crash with a raw RAID5 stack (with dd/fio
...).

It seems I need to stack filesystems to help reproduce it:

Here is a configuration test, command lines to explain (the way I'm able to
reproduce the crash). Everything is done in dom0.
- mdadm --create /dev/md10 --raid-devices=3 --level=5 /dev/sdc1 /dev/sdd1
/dev/sde1
- mkfs.btrfs /dev/md10
- mkdir /tmp/btrfs /mnt/XenVM /tmp/ext4
- mount /dev/md10 /tmp/btrfs
- btrfs subvolume create /tmp/btrfs/XenVM
- umount /tmp/btrfs
- mount /dev/md10 /mnt/XenVM -osubvol=XenVM
- truncate /mnt/XenVM/VMTestFile.dat -s 800G
- mkfs.ext4 /mnt/XenVM/VMTestFile.dat
- mount /mnt/XenVM/VMTestFile.dat /tmp/ext4

-> Doing this, doesn't seem to crash the kernel :
fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite
--rwmixwrite=95 --bs=1M --direct=1 --size=80G --numjobs=8 --runtime=600
--group_reporting --filename=/mnt/XenVM/Fio.dat

-> Doing this, is crashing the kernel in a few minutes :
fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randwrite
--rwmixwrite=95 --bs=1M --direct=1 --size=80G --numjobs=8 --runtime=600
--group_reporting --filename=/tmp/ext4/ext4.dat

Note : --direct=1 or --direct=0 doesn't seem to change the behaviour. Also
having the raid 5 stack re-synchronizing or already synchronized, doesn't
change the behaviour.

Here another "crash" : http://pastebin.com/uqLzL4fn
I'm trying to reproduce, but no success. So
ext4->btrfs->raid5, crash
btrfs->raid5, no crash
right? does subvolume matter? When you create the raid5 array, does adding
'--assume-clean' option change the behavior? I'd like to narrow down the issue.
If you can capture the blktrace to the raid5 array, it would be great to hint
us what kind of IO it is.
Yes Correct.
The subvolume doesn't matter.
-- assume-clean doesn't change the behaviour.
Don't forget that the system needs to be running on xen to crash, without (on native kernel) it doesn't crash (or at least, I was not able to make it crash).
Regarding your patch, I can't find it. Is it the one sent by Konstantin
Khlebnikov ?
Right.
It doesn't help :(. Maybe the crash is happening a little bit later.

Do you want the "ext4.dat" fio file ? It will be really difficult for me to
provide it to you as I've only a poor ADSL network connection.
Not necessary.

Thanks,
Shaohua

Thanks for your help,

MasterPrenium

Le 04/01/2017 à 23:30, Shaohua Li a écrit :
On Fri, Dec 23, 2016 at 07:25:56PM +0100, MasterPrenium wrote:
Hello Guys,

I've having some trouble on a new system I'm setting up. I'm getting a kernel BUG message, seems to be related with the use of Xen (when I boot the system _without_ Xen, I don't get any crash).
Here is configuration :
- 3x Hard Drives running on RAID 5 Software raid created by mdadm
- On top of it, DRBD for replication over another node (Active/passive cluster)
- On top of it, a BTRFS FileSystem with a few subvolumes
- On top of it, XEN VMs running.

The BUG is happening when I'm making "huge" I/O (20MB/s with a rsync for example) on the RAID5 stack.
I've to reset system to make it work again.
what did you mean 'huge' I/O (20M/s)? Is it possible you can reproduce the
issue with a raw raid5 raid? It would be even better if you can give me a fio
job file with the issue, so I can easily debug it.

also please check if upstream patch (e8d7c33 md/raid5: limit request size
according to implementation limits) helps.

Thanks,
Shaohua