[QUESTION] workqueue: Reducing flush_workqueue() overhead for unbound workqueues

[Yao Kai]

Hi,

I am seeing significant flush_workqueue() overhead for an unbound
workqueue with a very small concurrency requirement.

The original XFS discussion is here:

https://lore.kernel.org/all/20260827033654.1172495-1-ranhongyun1@xxxxxxxxxx/T/#u

XFS creates its CIL push workqueue as:

        alloc_workqueue("xfs-cil/%s",
                WQ_FREEZABLE | WQ_MEM_RECLAIM | WQ_UNBOUND, 4, ...);

The CIL pipeline has at most four push works scheduled or running
concurrently. During a synchronous log force, XFS calls
flush_workqueue() before queueing the new push work. This completes
previously queued pushes and reduces the latency of waiting for the
target checkpoint.

Since commit 636b927eba5b ("workqueue: Make unbound workqueues to use
per-cpu pool_workqueues"), flush_workqueue_prep_pwqs() walks the
per-CPU pwqs of the workqueue. Commit 85f0d8e39aff ("workqueue: Reduce
expensive locks for unbound workqueue") substantially reduced the
number of pool lock operations, but the O(nr_cpu_ids) pwq traversal
remains.

On a 128-CPU, 4-NUMA-node system, with 16 threads performing sequential
writes followed by fsync, I observed:

Function                      per-CPU pwqs     per-NUMA pwqs
-------------------------    ---------------  --------------
xfs_fsync_flush_log            256-1000 us      32-256 us
flush_workqueue_prep_pwqs        2-64 us          2-8 us

Dave pointed out that removing it may increase scheduling variance and
long-tail latency, and may move ordering latency to journal I/O
completion. He also suggested that changing the XFS caller would be a
workaround for an infrastructure regression, as other low-concurrency
unbound workqueues could have the same problem.

Is there a recommended way to avoid this overhead for such case,
I would appreciate guidance on how this problem should be solved.
I'm happy to prototype and benchmark the suggested approach.

Thanks,
Yao Kai