Re: [PATCH v3] blk-mq: dispatch kworker to housekeeping CPU when hctx cpumask is empty

[Daniel Wagner]

On Fri, Oct 16, 2026 at 01:22:03AM -0400, Aaron Tomlin wrote:
> Commit a46c27026da1 ("blk-mq: don't schedule block kworker on isolated
> CPUs") excluded isolated CPUs from hctx->cpumask to prevent worker
> threads from executing block work on isolated CPUs. When all CPUs mapped
> to a hardware queue are isolated (i.e. blk_mq_hctx_empty_cpumask(hctx)
> evaluates to true), blk_mq_hctx_next_cpu() returns WORK_CPU_UNBOUND with
> the intention of offloading execution.
> 
> However, kblockd_workqueue is allocated as a per-CPU workqueue
> (WQ_PERCPU). When WORK_CPU_UNBOUND is queued on a per-CPU workqueue, the
> workqueue core resolves the target CPU via raw_smp_processor_id().
> Consequently, if a task executing on an isolated CPU submits I/O and
> triggers an asynchronous queue dispatch with zero delay (e.g. upon
> hardware queue congestion, tag/resource exhaustion returning
> BLK_STS_RESOURCE, or plugging), &hctx->run_work is queued directly onto
> the local isolated CPU's worker pool. This causes the local per-CPU
> high-priority kworker (e.g. "kworker/6:1H-kblockd") to be scheduled on
> the isolated CPU. In real-time environments, an active SCHED_FIFO workload
> will starve this SCHED_NORMAL worker, causing hardware dispatch to stall,
> in addition to introducing unwanted latency jitter on nohz_full cores,
> thereby defeating the original objective of commit a46c27026da1.
> 
> Introduce blk_mq_hctx_unbound_cpu() to select an online housekeeping CPU
> from housekeeping_cpumask(HK_TYPE_DOMAIN) via
> cpumask_any_and_distribute() when executing on an isolated CPU, and
> otherwise fall back to WORK_CPU_UNBOUND. Use this helper across the
> unbound fallback paths in blk_mq_hctx_next_cpu()
> 
> This ensures that:
>     1.  When executing on an isolated CPU, deferred queue work is
>         cleanly offloaded to an online housekeeping CPU across single-queue
>         devices, hardware queues with empty cpumasks, and offline CPU
>         fallback paths.
> 
>     2.  The per-CPU high-priority kworker on the isolated CPU is not
>         scheduled, avoiding priority inversion and latency jitter.
> 
>     3.  When executing on a non-isolated CPU, WORK_CPU_UNBOUND is
>         preserved, maintaining optimal local execution with zero CPU
>         bouncing.
> 
> Fixes: a46c27026da1 ("blk-mq: don't schedule block kworker on isolated CPUs")
> Cc: Ming Lei <ming.lei@xxxxxxxxxx>
> Cc: Jens Axboe <axboe@xxxxxxxxx>
> Cc: stable@xxxxxxxxxxxxxxx
> Signed-off-by: Aaron Tomlin <atomlin@xxxxxxxxxxx>
> ---
> Changes since v2:
> 
>  - Query housekeeping_cpumask(HK_TYPE_DOMAIN) instead of HK_TYPE_TIMER
>    to align with cpu_is_isolated(), workqueue affinity rules, and cpuset
>    isolated partitions
> 
>  - Query the housekeeping cpumask under guard(rcu)() to safely handle
>    dynamic cpuset updates without lockdep warnings
> 
>  - Validate that cpumask_any_and_distribute() returns a valid cpu ID
>    before returning, falling back cleanly to WORK_CPU_UNBOUND
> 
>  - Link to v2: https://lore.kernel.org/all/20261015020225.422107-1-atomlin@xxxxxxxxxxx/
> 
> Changes since v1:
> 
>  - Introduce the blk_mq_hctx_unbound_cpu() helper to steer deferred work
>    to an online housekeeping CPU only when executing on an isolated
>    core, preserving WORK_CPU_UNBOUND and local cache locality for
>    housekeeping cores
> 
>  - Apply blk_mq_hctx_unbound_cpu() across all unbound fallback paths in
>    blk_mq_hctx_next_cpu(), comprehensively covering single-queue devices
>    (nr_hw_queues == 1), empty hardware queue cpumasks, and the offline
>    CPU fallback path.
> 
>  - Elaborate in the commit log on the triggers for asynchronous queue
>    dispatch (e.g. tag exhaustion, BLK_STS_RESOURCE backpressure, and
>    plugging) and the SCHED_FIFO worker starvation hazard on isolated
>    cores
> 
>  - Link to v1: https://lore.kernel.org/lkml/20261012012905.267920-1-atomlin@xxxxxxxxxxx/

Looks reasonable and related to my patch as well

  https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-4-355156be27b2@xxxxxxxxxx

and I was able to create a blktests for it:

  https://lore.kernel.org/all/20261014-nvme-isolcpu-v1-0-2fd175a8ef00@xxxxxxxxxx

Would it possible to extend those tests or add a new one so we get this
also tested? And these two patches then should/could merge.