Re: [RFC PATCH] blk-mq: Clean up references when freeing rqs
From: John Garry
Date: Thu Dec 10 2020 - 07:24:23 EST
Hi Ming,
So I'll try an experiment today to prove your concern about
blk_mq_queue_tag_busy_iter(). Then look at possible solution which
builds on patch in $subject, and compare.
JFYI, I was able to trigger this:
[ 1651.324150]
==================================================================
[ 1651.331400] BUG: KASAN: use-after-free in bt_iter+0xa0/0x120
[ 1651.337054] Read of size 8 at addr ffff00108d589300 by task fio/3052
[ 1651.344891] CPU: 32 PID: 3052 Comm: fio Tainted: GW
5.10.0-rc4-64839-g2dcf1ee5054f #693
[ 1651.354281] Hardware name: Huawei Taishan 2280 /D05, BIOS Hisilicon
D05 IT21 Nemo 2.0 RC0 04/18/2018
[ 1651.363407] Call trace:
[ 1651.365861] dump_backtrace+0x0/0x2d0
[ 1651.369519] show_stack+0x18/0x68
[ 1651.372833] dump_stack+0x100/0x16c
[ 1651.376316] print_address_description.constprop.12+0x6c/0x4e8
[ 1651.382146] kasan_report+0x130/0x200
[ 1651.385801] __asan_load8+0x9c/0xd8
[ 1651.389284] bt_iter+0xa0/0x120
[ 1651.392419] blk_mq_queue_tag_busy_iter+0x2d8/0x540
[ 1651.397293] blk_mq_in_flight+0x80/0xb8
[ 1651.401121] part_stat_show+0xd8/0x238
[ 1651.404867] dev_attr_show+0x44/0x90
[ 1651.408439] sysfs_kf_seq_show+0x128/0x1c8
[ 1651.412530] kernfs_seq_show+0xa0/0xb8
[ 1651.416274] seq_read_iter+0x1ec/0x6a0
[ 1651.420019] seq_read+0x1d0/0x250
[ 1651.423331] kernfs_fop_read+0x70/0x330
[ 1651.427161] vfs_read+0xe4/0x250
[ 1651.430382] ksys_read+0xc8/0x178
[ 1651.433690] __arm64_sys_read+0x44/0x58
[ 1651.437524] el0_svc_common.constprop.2+0xc4/0x1e8
[ 1651.442309] do_el0_svc+0x90/0xa0
[ 1651.445619] el0_sync_handler+0x128/0x178
[ 1651.449623] el0_sync+0x158/0x180
[ 1651.454418] The buggy address belongs to the page:
[ 1651.459208] page:00000000140c0813 refcount:0 mapcount:0
mapping:0000000000000000 index:0x0 pfn:0x108d589
[ 1651.468683] flags: 0xbfffc0000000000()
[ 1651.472432] raw: 0bfffc0000000000 0000000000000000 ffffffff42150201
0000000000000000
[ 1651.480173] raw: 0000000000000000 0000000000000000 00000000ffffffff
0000000000000000
[ 1651.487909] page dumped because: kasan: bad access detected
[ 1651.494956] Memory state around the buggy address:
[ 1651.499744] ffff00108d589200: ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff
[ 1651.506960] ffff00108d589280: ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff
[ 1651.514176] >ffff00108d589300: ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff
[ 1651.521389] ^
[ 1651.524611] ffff00108d589380: ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff
[ 1651.531827] ffff00108d589400: ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff
[ 1651.539044]
==================================================================
[ 1651.546258] Disabling lock debugging due to kernel taint
john@ubuntu:~$
So I run fio on disk providing root partition and another disk, and
constantly change IO scheduler on other disk.
I did also add this delay to trigger in reasonable timeframe:
+++ b/block/blk-mq-tag.c
@@ -215,8 +215,11 @@ static bool bt_iter(struct sbitmap *bitmap,
unsigned int bitnr, void *data)
* We can hit rq == NULL here, because the tagging functions
* test and set the bit before assigning ->rqs[].
*/
- if (rq && rq->q == hctx->queue && rq->mq_hctx == hctx)
- return iter_data->fn(hctx, rq, iter_data->data, reserved);
+ if (rq) {
+ mdelay(50);
+ if (rq->q == hctx->queue && rq->mq_hctx == hctx)
+ return iter_data->fn(hctx, rq, iter_data->data, reserved);
+ }
return true;
}
Thanks,
John