Re: Fwd: 4.4-rc2 crash: block related
From: Jan Kara
Date: Wed Nov 25 2015 - 03:39:51 EST
On Wed 25-11-15 08:27:56, Mika Penttilä wrote:
> With recent block layer pull i see a 100% repeatable crash on boot while
> mounting roots (ext4 partition on eMMC, with cfq io scheduler).
Thanks for report! After some investigation I found out we allocate
elevator specific data in __get_request() only for non-flush requests. And
this is actually required since the flush machinery uses the space in
struct request for something else. Doh. So my patch is just wrong and not
easy to fix since at the time __get_request() is called we are not sure
whether the flush machinery will be used in the end. Jens, please revert
1b2ff19e6a957b1ef0f365ad331b608af80e932e. Thanks!
I'm somewhat surprised that you can reliably hit the race where flushing
gets disabled for the device just while the request is in flight. But I
guess during boot it makes some sense.
Honza
> 5.674294] Unable to handle kernel NULL pointer dereference at virtual
> address 00000004
> [ 5.682399] pgd = a8ca4000
> [ 5.685113] [00000004] *pgd=38a5e831, *pte=00000000, *ppte=00000000
> [ 5.691428] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
> [ 5.696830] Modules linked in: st_drv
> [ 5.700533] CPU: 1 PID: 221 Comm: mount Not tainted 4.4.0-rc2 #49
> [ 5.706631] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
> [ 5.713163] task: a88e2ac0 ti: a88d4000 task.ti: a88d4000
> [ 5.718578] PC is at cfq_init_prio_data+0x8/0xec
> [ 5.723206] LR is at cfq_insert_request+0x28/0x4f0
> [ 5.723211] pc : [<8024bf9c>] lr : [<8024e768>] psr: 600d0093
> [ 5.723211] sp : a88d5bc0 ip : 00000000 fp : a8ab5400
> [ 5.723219] r10: 00000001 r9 : a617f4c0 r8 : 80b6359c
> [ 5.723223] r7 : 80b62100 r6 : a873e200 r5 : a885ac30 r4 : 00000000
> [ 5.723226] r3 : a88d5bc0 r2 : a89106c0 r1 : 00000000 r0 : 00000000
> [ 5.723232] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM
> Segment user
> [ 5.723235] Control: 10c5387d Table: 38ca404a DAC: 00000055
> [ 5.723239] Process mount (pid: 221, stack limit = 0xa88d4210)
> [ 5.723242] Stack: (0xa88d5bc0 to 0xa88d6000)
> [ 5.723251] 5bc0: 00000000 a885ac30 a873e200 8024e768 a87c0000
> a885ac30 00000005 a88d4000
> [ 5.723257] 5be0: 80b6359c a617f4c0 00000001 8023817c ffffffff
> a89106c0 a885ac30 00000000
> [ 5.723263] 5c00: a89106c0 ffffffff a87c0000 8023654c 00000000
> 00000000 a8ab5400 a89106c0
> [ 5.723269] 5c20: 00000008 00001411 f0000000 80236680 a88d5c44
> a87c0168 a617f4c0 a81a45c0
> [ 5.723276] 5c40: 00000001 02400000 80b6359c a617f4c0 00000001
> 80231b04 a00d0013 0000000f
> [ 5.723282] 5c60: a617f4c0 a89106c0 00001411 f0000000 80b6359c
> a617f4c0 00000001 80110950
> [ 5.723288] 5c80: a617f4c0 00000001 00001411 80b6370c 80b6359c
> 80112490 a8b35c00 00000000
> [ 5.723295] 5ca0: 80b63658 801602e4 0205a9d9 00000000 a62b4738
> a8ab5400 a8b35c00 a8b36000
> [ 5.723301] 5cc0: 00000000 00000000 a8b36000 a8ab5400 a88d5e8c
> 80162644 a62621e8 800f7004
> [ 5.723307] 5ce0: a88d5e8c 806dd610 a62621e8 a617f4c0 a8b35c00
> a8b36000 00000001 80165480
> [ 5.723313] 5d00: 00000000 00000000 a88d5d58 a88d5d50 a87f2a90
> a88d5d54 01897158 800ec9dc
> [ 5.723319] 5d20: 00000000 00000002 00000000 a88d5dc8 00000001
> a88d5dc0 00000001 a6023000
> [ 5.723325] 5d40: a88d5d90 a88d5d88 a8887f10 a88d5d8c 01897158
> 800ec8d0 a8887f10 00000004
> [ 5.723332] 5d60: 00000000 a88d5dc0 a88d5dc0 a6029110 00000001
> a80fd000 a88d5d8c a8744800
> [ 5.723338] 5d80: 00000000 00000000 00000001 00000980 0000b67c
> 00000000 00000001 800bf478
> [ 5.723343] 5da0: a615e490 00000001 0000006c a8102db0 00000000
> 00000001 0000000a 00000001
> [ 5.723349] 5dc0: 0000ffff 00000000 00000000 00000000 0000002b
> a82ec200 80b6e735 00000004
> [ 5.723355] 5de0: 00000000 00000000 a8ab5400 00000000 a8b36264
> 00000000 001013d0 00000000
> [ 5.723361] 5e00: 00000001 00000000 a8b36000 00000000 00001000
> a8b35e88 00000000 00000000
> [ 5.723366] 5e20: 00000000 00000000 a8ab5594 00000000 80be3e54
> 00000000 00000000 00000000
> [ 5.723372] 5e40: 00000000 00004003 00000000 80b70288 01897158
> 8025e5bc a6298e00 a88d5e6c
> [ 5.723378] 5e60: 3b9aca00 00000009 a6298e00 a6298e74 a8b35c00
> a6298e00 00000083 00000000
> [ 5.723384] 5e80: 00000000 80b70288 01897158 800e6324 a6298e00
> 800c0050 62636d6d 70306b6c
> [ 5.723391] 5ea0: a8000035 800d0013 00000004 80be3e2c a8dca80e
> 00000000 00000001 8015f030
> [ 5.723397] 5ec0: a8dca800 00000000 80b70288 80b70288 80b6aeb0
> 8015f048 801636d8 a8ab1a48
> [ 5.723403] 5ee0: 01897158 800e6f14 00000000 a8dca800 a8ab19c0
> a8dca800 00000000 80b70288
> [ 5.723409] 5f00: 00000000 800febbc 00000000 00000020 00000000
> a8dca800 a8dca840 80101a14
> [ 5.723416] 5f20: 00000000 80b60be0 a8001f00 024000c0 000088c5
> 800df23c 007fffff a8dca800
> [ 5.723421] 5f40: a87f2a90 a6138cc0 c0ed0000 a8dca800 0000000f
> 00000000 0000000f a8dca840
> [ 5.723428] 5f60: a8dca800 00000000 018971a0 c0ed0000 a88d4000
> 00000000 01897158 801027e4
> [ 5.723434] 5f80: 00000000 28936a1b 563c86d0 00000000 00000000
> 76f35688 c0ed0000 00000015
> [ 5.723440] 5fa0: 8000f6a4 8000f500 00000000 76f35688 01897188
> 018971a0 01897158 c0ed0000
> [ 5.723447] 5fc0: 00000000 76f35688 c0ed0000 00000015 018971a0
> 01897188 76f36dac 01897158
> [ 5.723453] 5fe0: 76e56dc0 7eedcc30 76f09e70 76e56dd0 600d0010
> 01897188 00000000 00000000
> [ 5.723473] [<8024bf9c>] (cfq_init_prio_data) from [<8024e768>]
> (cfq_insert_request+0x28/0x4f0)
> [ 5.723484] [<8024e768>] (cfq_insert_request) from [<8023817c>]
> (blk_queue_bio+0x254/0x260)
> [ 5.723500] [<8023817c>] (blk_queue_bio) from [<8023654c>]
> (generic_make_request+0xcc/0x17c)
> [ 5.723510] [<8023654c>] (generic_make_request) from [<80236680>]
> [ 5.723527] [<80236680>] (submit_bio) from [<80110950>]
> (submit_bh_wbc+0x10c/0x144)
> [ 5.723537] [<80110950>] (submit_bh_wbc) from [<80112490>]
> (__sync_dirty_buffer+0x90/0x114)
> [ 5.723550] [<80112490>] (__sync_dirty_buffer) from [<801602e4>]
> (ext4_commit_super+0x1c8/0x290)
> [ 5.723561] [<801602e4>] (ext4_commit_super) from [<80162644>]
> (ext4_setup_super+0xe8/0x1d0)
> [ 5.723569] [<80162644>] (ext4_setup_super) from [<80165480>]
> (ext4_fill_super+0x1da8/0x32b8)
> [ 5.723580] [<80165480>] (ext4_fill_super) from [<800e6324>]
> (mount_bdev+0x168/0x190)
> [ 5.723588] [<800e6324>] (mount_bdev) from [<8015f048>]
> (ext4_mount+0x18/0x20)
> [ 5.723596] [<8015f048>] (ext4_mount) from [<800e6f14>]
> (mount_fs+0x14/0xa4)
> [ 5.723604] [<800e6f14>] (mount_fs) from [<800febbc>]
> (vfs_kern_mount+0x4c/0xf4)
> [ 5.723614] [<800febbc>] (vfs_kern_mount) from [<80101a14>]
> (do_mount+0x1a0/0xc24)
> [ 5.723622] [<80101a14>] (do_mount) from [<801027e4>]
> (SyS_mount+0x74/0xa0)
> [ 5.723635] [<801027e4>] (SyS_mount) from [<8000f500>]
> (ret_fast_syscall+0x0/0x34)
> [ 5.723642] Code: e3a00000 e8bd8070 e92d4070 e1a0300d (e5902004)
> [ 5.723647] ---[ end trace 4087b61c23235e2b ]---
> [ 5.723651] Kernel panic - not syncing: Fatal exception
> ---
>
>
> Cause seems to be this commit :
>
> commit 1b2ff19e6a957b1ef0f365ad331b608af80e932e
> Author: Jan Kara <jack@xxxxxxxx>
> Date: Thu Nov 12 14:25:52 2015 +0100
>
> blk-flush: Queue through IO scheduler when flush not required
>
>
>
> git show 1b2ff19e6a957b1ef0f365ad331b608af80e932e:
>
> diff --git a/block/blk-flush.c b/block/blk-flush.c
> index 9c423e5..c81d56e 100644
> --- a/block/blk-flush.c
> +++ b/block/blk-flush.c
> @@ -422,7 +422,7 @@ void blk_insert_flush(struct request *rq)
> if (q->mq_ops) {
> blk_mq_insert_request(rq, false, false, true);
> } else
> - list_add_tail(&rq->queuelist, &q->queue_head);
> + q->elevator->type->ops.elevator_add_req_fn(q, rq);
> return;
> }
>
>
>
> With this reverted it boots normally.
>
> Thanks,
> Mika
>
>
>
--
Jan Kara <jack@xxxxxxxx>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/