Re: [PATCH 3/3] workqueue: Make flush_workqueue() visit only pwqs active since the last flush

[Breno Leitao]

On Tue, Sep 01, 2026 at 11:09:29AM -1000, Tejun Heo wrote:
> 636b927eba5b ("workqueue: Make unbound workqueues to use per-cpu
> pool_workqueues") fixed unbound workqueue scalability on large machines but
> made flush_workqueue() walk one pwq per possible CPU, cycling each pool
> lock, up to three times per flush. 85f0d8e39aff ("workqueue: Reduce
> expensive locks for unbound workqueue") coalesced same-pool locks for
> unbound workqueues but the walk remains. Yao Kai reported the XFS CIL
> workqueue, flushed on every log force, spending up to 64us per flush in the
> walk on a 128-CPU machine.
> 
> Idle pwqs can't just be skipped: the lock-and-advance of every pwq's work
> color is what keeps a concurrently queued work item from being stamped with
> a retired color unseen by the flusher, which would let a later flush return
> before it finishes.
> 
> Track active pwqs instead. Each workqueue gets a wq_flush_pnode per node
> with a lock, a mirror of wq->work_color and a list of pwqs. Queueing to an
> off-list pwq syncs pwq->work_color from the mirror and adds it under
> fpn->lock, and flush_workqueue_prep_pwqs() advances the mirror and splices
> the list in one fpn->lock section per node before visiting the pwqs under
> pool->lock as before. That replaces the fence: a racing queueing either gets
> its pwq on the list before the splice or stamps the advanced color. A pwq
> stays on the list until a visit finds nothing in flight, so a cascade arming
> an older color still finds it, barriers need no separate add as the work
> item they follow keeps the pwq on the list, and pwq_release_workfn() removes
> a released pwq under wq->mutex.
> 
> On a 192-CPU 2-node machine, flushing an idle per-cpu workqueue goes from
> 40k to 4.2M per second and an idle unbound one from 450k to 4.3M. 16 threads
> each queueing a work item and flushing, go from 30k to 110k flushes per
> second on a per-cpu workqueue and 60k to 180k on an unbound one. Dense
> flushes with every pwq active are unchanged on per-cpu workqueues and 15-20%
> slower on unbound ones. The queue path is unchanged.
> 
> Reported-by: Yao Kai <yaokai34@xxxxxxxxxx>
> Link: https://lore.kernel.org/all/0a030145-c108-4365-ba2d-ac1973a1e352@xxxxxxxxxx/
> Signed-off-by: Tejun Heo <tj@xxxxxxxxxx>

Tested-by: Breno Leitao <leitao@xxxxxxxxxx>