Re: [RFC PATCH 0/2] Add queue_*() functions and prefer per-cpu workqueue and flag

[Marco Crivellari]

On Wed, May 6, 2026 at 3:40 PM Breno Leitao <leitao@xxxxxxxxxx> wrote:
>
> On Tue, May 05, 2026 at 10:18:49AM -1000, Tejun Heo wrote:
> > (cc'ing Breno)

Thanks Tejun and Breno, for your feedbacks.

> > On Tue, May 05, 2026 at 06:16:56PM +0200, Marco Crivellari wrote:
> > > Actually schedule_work() and schedule_work_on() enqueue works using
> > > system_percpu_wq. The function name doesn't suggest it, on top of that,
> > > only the per-cpu version is present.
> >
> > I was hoping to just retire schedule_work[_on]() and let people use e.g.
> > system_percpu_wq directly. Is that too verbose for casual users?
>
> I think schedule_work() doesn't help much, and makes the system a bit harder to
> understand. When I started reading this code, I would have preferred to see
> queue_work(system_percpu_wq, work) instead of schedule_work(work).
>
> In fact, I suspect this patchset exists partly because we have the
> schedule_work() helper.

Yes, correct. Perhaps retiring schedule_work[_on](), as Tejun
suggested, would be the easiest way indeed.

I proposed this in light of our next step (which I would say is the
first): ensuring that every schedule_work() really needs to use the
per-cpu workqueue and offloads work that can be unbound to the unbound
workqueue.

So, either we're going to have an "unbound" version or we use
queue_work() directly that sounds good to me. I guess retire - in
future - schedule_work[_on]() would be cleaner: so that users must
also specify the workqueue they really need to use.

> > > Because of that, the following changes are introduced:
> > >
> > > - queue_{bound|unbound}_work() as future replacement of schedule_work()
> >
> > If we do this, I think "percpu" is a lot clearer than "bound". percpu <->
> > (nothing) combination would be nice eventually but maybe that's too
> > confusing now. Does percpu <-> unbound combination sound weird?
>
> Would percpu <-> global sound less weird?

Now that I read your inputs, if we rename, perhaps we can keep the
current abbreviation for unbound ("dfl") to avoid introducing
something new.

What do you both think about:

- queue_percpu_work()
- queue_dfl_work()

?

They somehow reflect the newly introduced system_dfl_wq and system_percpu_wq.

> > > The Workqueue API currently do not distinguish between use case where
> > > locality is important for correctness and where is important for
> > > efficiency.
>
> If you enqueue work to system_unbound_wq with the default affinitization, you
> already get locality (WQ_AFFN_CACHE groups CPUs sharing the same LLC). This is
> the way to say that locality is important for efficiency, anbd the WQ_AFFN_CPU
> is the way to specify that locality is important for correctness.
>
> On top of that, WQ_AFFN_SYSTEM is a way to specify that locality is not
> necessary at all.
>
> Also, how WQ_PREFER_PERCPU behaves differently from WQ_AFFN_CPU?

Let me share where this was discussed a year ago:

https://lore.kernel.org/all/Z79E_gbWm9j9bkfR@xxxxxxxxxxxxxxx/

Perhaps - likely - I haven't understood the WQ_PREFER_PERCPU proposal
here; I thought it was a workqueue flag, to be used like WQ_PERCPU or
WQ_UNBOUND.
Reading Tejun's reply is also clearer now.

Anyhow, this idea is based on customer reports I've seen previously.
We noticed that with certain workloads, specific per-cpu work creates
noise on isolated CPUs. With a flag like that we can identify which
workqueues prefer to be per-cpu and *not* for correctness. This allows
using a boot parameter / sysctl, for example, to keep those workqueues
affined only to housekeeping CPUs.

Of course, if we can achieve the same with a system workqueue (like
system_prefer_percpu_wq), that would also be fine. I think it would be
way easier, it should be similar to what we're doing with
system_power_efficient_wq [1].

Tejun, Breno (and others), what do you think? Bad idea? :-)

Thanks!

- [1] https://elixir.bootlin.com/linux/v7.0.1/source/kernel/workqueue.c#L7907

--

Marco Crivellari

SUSE Labs