Re: [PATCH] blk-mq: Allow complete locally if capacities are different

[Christian Loehle]

On 8/28/24 13:13, Bart Van Assche wrote:
> On 8/28/24 7:49 AM, Manish Pandey wrote:
>> 'Commit af550e4c9682 ("block/blk-mq: Don't complete locally if
>> capacities are different")' enforces to complete the request locally
>> only if the submission and completion CPUs have same capacity.
>>
>> To have optimal IO load balancing or to avoid contention b/w submission
>> path and completion path, user may need to complete IO request of large
>> capacity CPU(s) on Small Capacity CPU(s) or vice versa.
>>
>> Hence introduce a QUEUE_FLAG_ALLOW_DIFF_CAPACITY blk queue flag to let
>> user decide if it wants to complete the request locally or need an IPI
>> even if the capacity of the requesting and completion queue is different.
>> This gives flexibility to user to choose best CPU for their completion
>> to give best performance for their system.

Why have the flag be
QUEUE_FLAG_ALLOW_DIFF_CAPACITY
instead of just
QUEUE_FLAG_SAME_LLC
so it isn't as HMP specifically-worded?
(And of course then having QUEUE_FLAG_SAME_COMP be strictly
stronger than QUEUE_FLAG_SAME_LLC. On !HMP they are equal.)
That would also answer Bart's question below how this is
different to rq_affinity=0.

> 
> I think that the following is missing from the above description:
> - Mentioning that this is for an unusual interrupt routing technology
>   (SoC sends the interrupt to another CPU core than what has been
>    specified in the smp_affinity mask).

FWIW on !mcq that doesn't have to be the case.

> - An explanation why the desired effect cannot be achieved by changing
>   rq_affinity into 0.
> 
>>   block/blk-mq-debugfs.c |  1 +
>>   block/blk-mq.c         |  3 ++-
>>   block/blk-sysfs.c      | 12 ++++++++++--
>>   include/linux/blkdev.h |  1 +
>>   4 files changed, 14 insertions(+), 3 deletions(-)
> 
> Since the semantics of a sysfs attribute are modified,
> Documentation/ABI/stable/sysfs-block should be updated.
> 
> Thanks,
> 
> Bart.
>