Re: [PATCH 1/2] block: default to rq_affinity=2 for blk-mq

[Jens Axboe]

On 09/09/2014 06:18 PM, Robert Elliott wrote:
> From: Robert Elliott <elliott@xxxxxx>
> 
> One change introduced by blk-mq is that it does all
> the completion work in hard irq context rather than
> soft irq context.
> 
> On a 6 core system, if all interrupts are routed to
> one CPU, then you can easily run into this:
> * 5 CPUs submitting IOs
> * 1 CPU spending 100% of its time in hard irq context
> processing IO completions, not able to submit anything
> itself
> 
> Example with CPU5 receiving all interrupts:
>    CPU usage:   CPU0   CPU1   CPU2   CPU3   CPU4   CPU5
>         %usr:   0.00   3.03   1.01   2.02   2.00   0.00
>         %sys:  14.58  75.76  14.14   4.04  78.00   0.00
>         %irq:   0.00   0.00   0.00   1.01   0.00 100.00
>        %soft:   0.00   0.00   0.00   0.00   0.00   0.00
> %iowait idle:  85.42  21.21  84.85  92.93  20.00   0.00
>        %idle:   0.00   0.00   0.00   0.00   0.00   0.00
> 
> When the submitting CPUs are forced to process their own
> completion interrupts, this steals time from new
> submissions and self-throttles them.
> 
> Without that, there is no direct feedback to the
> submitters to slow down.  The only feedback is:
> * reaching max queue depth
> * lots of timeouts, resulting in aborts, resets, soft
>   lockups and self-detected stalls on CPU5, bogus
>   clocksource tsc unstable reports, network
>   drop-offs, etc.
> 
> The SCSI LLD can set affinity_hint for each of its
> interrupts to request that a program like irqbalance
> route the interrupts back to the submitting CPU.
> The latest version of irqbalance ignores those hints,
> though, instead offering an option to run a policy
> script that could honor them. Otherwise, it balances
> them based on its own algorithms. So, we cannot rely
> on this.
> 
> Hardware might perform interrupt coalescing to help,
> but it cannot help 1 CPU keep up with the work
> generated by many other CPUs.
> 
> rq_affinity=2 helps by pushing most of the block layer
> and SCSI midlayer completion work back to the submitting
> CPU (via an IPI).
> 
> Change the default rq_affinity=2 under blk-mq
> so there's at least some feedback to slow down the
> submitters.

I don't think we should do this generically. For "sane" devices with
multiple completion queues, and with proper affinity setting in the
driver, this is going to be a loss.

So lets not add it to QUEUE_FLAG_MQ_DEFAULT, but we can make it default
for nr_hw_queues == 1. I think that would be way saner.

-- 
Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/