Re: [blk] 8c5035dfbb: fio.read_iops -10.6% regression

From: Feng Tang
Date: Thu Oct 13 2022 - 03:42:46 EST


On Sun, Oct 09, 2022 at 05:32:34PM +0800, Yu Kuai wrote:
> Hi,
>
> 在 2022/10/09 16:43, Ming Lei 写道:
> > On Sat, Oct 08, 2022 at 04:00:10PM +0800, Yu Kuai wrote:
> >> Hi,
> >>
> >> 在 2022/10/08 10:50, kernel test robot 写道:
> >>> Greeting,
> >>>
> >>> FYI, we noticed a -10.6% regression of fio.read_iops due to commit:
> >>
> >> I don't know how this is working but I'm *sure* this commit won't affect
> >
> > Looks it is wrong to move
> >
> > wbt_set_write_cache(q, test_bit(QUEUE_FLAG_WC, &q->queue_flags));
> >
> > before rq_qos_add() in wbt_init().
> >
> > Without adding wbt rq_qos, wbt_set_write_cache is just a nop.
>
> Yes, I got it now, I'm being foolish here.
>
> I missed that "rwb->wc" is got by rq_qos in wbt_set_write_cache(), which
> is NULL before rq_qos_add(). By the way, it's interesting that how read
> performance is affected, I still don't know why yet...

Indeed, we are confused too. So we did some further check, and found
it could be related with the less calls of wake_up_all(), due to the
'rwb->wc' value changed.

I'm not familiar with the block layer and VFS, and just checked the
'blk-wbt.c'. Before commit 8c5035dfbb, the 'rwb->wc' is 0 in 0Day's
test env, while it's 1 after the commit.

Inside wbt_rqw_done(), 'rwb->wc' be used to judge whether to wakeup
other waiters in system, so we add some debug code to check the
wakeup and skip-wakeup counter:

----------------------------------------------------------------
@@ -130,6 +130,8 @@ static void wbt_rqw_done(struct rq_wb *rwb, struct rq_wait *rqw,
/*
* Don't wake anyone up if we are above the normal limit.
*/
- if (inflight && inflight >= limit)
+ if (inflight && inflight >= limit) {
+ skip_wakeup++;
return;
+ }

if (wq_has_sleeper(&rqw->wait)) {
int diff = limit - inflight;

- if (!inflight || diff >= rwb->wb_background / 2)
+ if (!inflight || diff >= rwb->wb_background / 2) {
+ wakeup++;
wake_up_all(&rqw->wait);
+ }
----------------------------------------------------------------

And after the fio task, the 'skip_wakeup' number is much bigger
after the patch:

before patch:
422.274394: wbt_rqw_done: wakeup_skip=19408 wakup_all=1944759

after patch:
433.753345: wbt_rqw_done: wakeup_skip=2090585 wakup_all=13630

Hope this can help the root causing.

Thanks,
Feng