Re: [RFC] blk-mq: clean up the hctx restart

[jianchao.wang]

Hi Ming

On 08/01/2018 04:58 PM, Ming Lei wrote:
> On Wed, Aug 01, 2018 at 10:17:30AM +0800, jianchao.wang wrote:
>> Hi Ming
>>
>> Thanks for your kindly response.
>>
>> On 07/31/2018 02:16 PM, Ming Lei wrote:
>>> On Tue, Jul 31, 2018 at 01:19:42PM +0800, jianchao.wang wrote:
>>>> Hi Ming
>>>>
>>>> On 07/31/2018 12:58 PM, Ming Lei wrote:
>>>>> On Tue, Jul 31, 2018 at 12:02:15PM +0800, Jianchao Wang wrote:
>>>>>> Currently, we will always set SCHED_RESTART whenever there are
>>>>>> requests in hctx->dispatch, then when request is completed and
>>>>>> freed the hctx queues will be restarted to avoid IO hang. This
>>>>>> is unnecessary most of time. Especially when there are lots of
>>>>>> LUNs attached to one host, the RR restart loop could be very
>>>>>> expensive.
>>>>>
>>>>> The big RR restart loop has been killed in the following commit:
>>>>>
>>>>> commit 97889f9ac24f8d2fc8e703ea7f80c162bab10d4d
>>>>> Author: Ming Lei <ming.lei@xxxxxxxxxx>
>>>>> Date:   Mon Jun 25 19:31:48 2018 +0800
>>>>>
>>>>>     blk-mq: remove synchronize_rcu() from blk_mq_del_queue_tag_set()
>>>>>
>>>>>
>>>>
>>>> Oh, sorry, I didn't look into this patch due to its title when iterated the mail list,
>>>> therefore I didn't realize the RR restart loop has already been killed. :)
>>>>
>>>> The RR restart loop could ensure the fairness of sharing some LLDD resource,
>>>> not just avoid IO hung. Is it OK to kill it totally ?
>>>
>>> Yeah, it is, also the fairness might be improved a bit by the way in
>>> commit 97889f9ac24f8d2fc, especially inside driver tag allocation
>>> algorithem.
>>>
>>
>> Would you mind to detail more here ?
>>
>> Regarding the driver tag case:
>> For example:
>>
>> q_a         q_b        q_c       q_d
>> hctx0       hctx0      hctx0     hctx0
>>
>>                  tags
>>
>> Total number of tags is 32
>> All of these 4 q are active.
>>
>> So every q has 8 tags.
>>
>> If all of these 4 q have used up their 8 tags, they have to wait.
>>
>> When part of the in-flight requests q_a are completed, tags are freed.
>> but the __sbq_wake_up doesn't wake up the q_a, it may wake up q_b.
> 
> 1) in case of IO scheduler
> q_a should be waken up because q_a->hctx0 is added to one wq of the tags if
> no tag is available, see blk_mq_mark_tag_wait().
> 
> 2) in case of none scheduler
> q_a should be waken up too, see blk_mq_get_tag().
> 
> So I don't understand why you mentioned that q_a can't be waken up.

There are multiple sbq_wait_states in one sbitmap_queue and __sbq_wake_up
will only wake up the waiters on one of them one time. Please refer to __sbq_wake_up.

> 
>> However, due to the limits in hctx_may_queue, q_b still cannot get the
>> tags. The RR restart also will not wake up q_a.
>> This is unfair for q_a.
>>
>> When we remove RR restart fashion, at least, the q_a will be waked up by
>> the hctx restart.
>> Is this the improvement of fairness you said in driver tag allocation ?
> 
> I mean the fairness is totally covered by the general tag allocation
> algorithm now, which is sort of FIFO style because of waitqueue, but RR
> restart wakes up queue in the order of request queue.

Yes, I got your point.

> 
>>
>> Think further, it seems that it only works for case with io scheduler.
>> w/o io scheduler, tasks will wait in blk_mq_get_request. restart hctx will
>> not work there.
> 
> When one tag is freed, the sbitmap queue will be waken up, then some of
> allocation may be satisfied, this way works for both IO sched and none.
> 
> Thanks, 
> Ming
>