Re: [PATCH] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Aaron Tomlin]

On Tue, Oct 13, 2026 at 07:28:45AM -0500, Ming Lei wrote:
> No.
> 
> For example, schedule_work() is queue_work_on(WORK_CPU_UNBOUND, system_percpu_wq),
> a WQ_PERCPU workqueue, so every schedule_work() from an isolated CPU runs
> on that CPU. blk-mq isn't special here; this is the general tradeoff of
> submitting IO from an isolated CPU.

Fair point regarding schedule_work(). I concede that mechanically,
kblockd_workqueue is not unique in accepting WORK_CPU_UNBOUND on a per-CPU
workqueue. schedule_work() indeed directs WORK_CPU_UNBOUND to
system_percpu_wq, which resolves to raw_smp_processor_id() within
__queue_work().

However, the crucial distinction lies in the semantic contract and intent.
Callers of schedule_work() seek local (or best-effort local) execution;
there is neither an assumption nor a guarantee that schedule_work() avoids
isolated CPUs.

By contrast, blk-mq has an explicit architectural contract to avoid running
block kworkers on isolated CPUs. In commit a46c27026da1 ("blk-mq: don't
schedule block kworker on isolated CPUs"), the intent was to deliberately
excluded isolated CPUs from hctx->cpumask precisely because:

    "blk-mq kworker shouldn't be scheduled on isolated CPUs."

Furthermore, blk_mq_run_hw_queue() explicitly honours this policy:

    if (async || !cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)) {
        blk_mq_delay_run_hw_queue(hctx, 0);
        return;
    }

When executing on an isolated core (!cpumask_test_cpu(...)), blk-mq
deliberately refrains from inline dispatch specifically to offload the work
asynchronously via blk_mq_delay_run_hw_queue().

When all CPUs mapped to that hardware queue are isolated
(blk_mq_hctx_empty_cpumask(hctx)), commit a46c27026da1 added:

    /* Switch to unbound if no allowable CPUs in this hctx */
    if (hctx->queue->nr_hw_queues == 1 || blk_mq_hctx_empty_cpumask(hctx))
        return WORK_CPU_UNBOUND;

The express intention of that change was to switch away from the isolated
CPUs. However, because kblockd_workqueue is WQ_PERCPU, returning
WORK_CPU_UNBOUND inadvertently resolves to raw_smp_processor_id(), queuing
the work right back onto the isolated CPU that blk-mq just went out of its
way to avoid.

That is why blk-mq is in an unusual position; it intended to avoid isolated
cores, but relied on WORK_CPU_UNBOUND to achieve that on a bound per-CPU
workqueue where that mechanism cannot take effect.

> You already disturb isolated CPUs by submitting IO via syscall in this
> case, so why does it matter here to dispatch on same isolated CPU one more
> time?

There is a fundamental difference between synchronous syscall execution and
asynchronous background kworker execution:

    1.  When an isolated user task issues a synchronous I/O request (e.g.
        via write(2)), execution occurs synchronously within the task's own
        context and CPU accounting. The application remains in control of
        when this overhead is incurred.

    2.  When dispatch must be deferred (e.g. BLK_STS_RESOURCE), blk-mq
        queues &hctx->run_work. A background kworker runs under
        SCHED_NORMAL. On isolated cores, workloads typically execute under
        real-time scheduling classes (SCHED_FIFO, SCHED_RR, SCHED_DEADLINE)
        or tight polling loops. A SCHED_FIFO task executing on that core
        will completely starve the local kworker. Consequently,
        blk_mq_run_work_fn() never runs, hardware dispatch stalls, pending
        completions cannot occur, and the I/O pipeline hangs indefinitely.

    3.  Even where starvation is avoided (e.g. via the EEVDF scheduler's
        Fair server or cooperative yielding), queuing bound work on a
        nohz_full isolated core forces the scheduler tick to re-activate,
        injecting unwanted jitter and defeating deterministic isolation
        guarantees.

Steering deferred queue dispatch to an online housekeeping CPU via
housekeeping_any_cpu(HK_TYPE_TIMER) sidesteps these pitfalls entirely.
The kworker executes promptly on a housekeeping core without priority
inversion, hardware dispatch proceeds unhindered, and the isolated CPU's
low-latency domain is preserved.


Kind regards,
-- 
Aaron Tomlin