Re: bug in tag handling in blk-mq?

From: Jens Axboe
Date: Tue May 08 2018 - 21:09:17 EST

On 5/8/18 3:19 PM, Jens Axboe wrote:
> On 5/8/18 2:37 PM, Jens Axboe wrote:
>> On 5/8/18 10:42 AM, Mike Galbraith wrote:
>>> On Tue, 2018-05-08 at 08:55 -0600, Jens Axboe wrote:
>>>> All the block debug files are empty...
>>> Sigh. Take 2, this time cat debug files, having turned block tracing
>>> off before doing anything else (so trace bits in dmesg.txt should end
>>> AT the stall).
>> OK, that's better. What I see from the traces:
>> - You have regular IO and some non-fs IO (from scsi_execute()). This mix
>> may be key.
>> - sdd has nothing pending, yet has 6 active waitqueues.
>> I'm going to see if I can reproduce this. Paolo, what kind of attempts
>> to reproduce this have you done?
> No luck so far. Out of the patches you referenced, I can only find the
> shallow depth change, since that's in the parent of this email. Can
> you send those as well?
> Perhaps also expand a bit on exactly what you are running. File system,
> mount options, etc.

Alright, I managed to reproduce it. What I think is happening is that
BFQ is limiting the inflight case to something less than the wake
batch for sbitmap, which can lead to stalls. I don't have time to test
this tonight, but perhaps you can give it a go when you are back at it.
If not, I'll try tomorrow morning.

If this is the issue, I can turn it into a real patch. This is just to
confirm that the issue goes away with the below.

diff --git a/lib/sbitmap.c b/lib/sbitmap.c
index e6a9c06ec70c..94ced15b6428 100644
--- a/lib/sbitmap.c
+++ b/lib/sbitmap.c
@@ -272,6 +272,7 @@ EXPORT_SYMBOL_GPL(sbitmap_bitmap_show);

static unsigned int sbq_calc_wake_batch(unsigned int depth)
+#if 0
unsigned int wake_batch;

@@ -284,6 +285,9 @@ static unsigned int sbq_calc_wake_batch(unsigned int depth)
wake_batch = max(1U, depth / SBQ_WAIT_QUEUES);

return wake_batch;
+ return 1;

int sbitmap_queue_init_node(struct sbitmap_queue *sbq, unsigned int depth,

Jens Axboe