Re: [PATCH 0/4] sched/numa: Add per-process automatic NUMA balancing control

[Yafang Shao]

On Fri, Oct 16, 2026 at 1:51 PM Chen Yu <yu.c.chen@xxxxxxxxx> wrote:
>
> On Fri, Oct 16, 2026 at 10:53:43AM +0800, Yafang Shao wrote:
> > On Fri, Oct 16, 2026 at 3:58 AM Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx> wrote:
> > >
> > > On Tue, 2026-10-13 at 17:31 +0800, Yafang Shao wrote:
> > > > On Tue, Oct 13, 2026 at 3:52 PM Chen Yu <yu.c.chen@xxxxxxxxx> wrote:
> > > > >
> > > > > On Tue, Sep 08, 2026 at 08:24:42PM +0800, Li Zhe wrote:
> > > > > > Automatic NUMA balancing is controlled globally through the
> > > > > > kernel.numa_balancing sysctl and can be influenced indirectly through
> > > > > > memory policy. That works well as a system default, but it is too coarse
> > > > > > for workloads where a launcher wants most processes to use the default
> > > > > > policy while a selected process opts out because it already manages NUMA
> > > > > > placement or cannot afford the sampling overhead.
> > > > > >
> > > > > > A recent per-cgroup proposal tried to address this by adding a
> > > > > > cgroup-local enable knob plus a NUMA_BALANCING_CGROUP mode where
> > > > > > balancing was disabled by default and then enabled for selected cgroups
> > > > > > [1]. The discussion also noted that automatic NUMA balancing already
> > > > > > operates at task/process granularity, that the knob is not a resource
> > > > > > control attribute, and that task/process controls such as sched_setattr()
> > > > > > or prctl() may be a better fit while cgroups and cpusets continue to
> > > > > > describe workload grouping and placement domains.
> > > > > >
> > > > > > There is also prior art from 2023 prctl() proposals [2][3]. Those
> > > > > > versions used a per-mm mode with disabled/enabled/default states, added
> > > > > > NumaB_mode to /proc/<pid>/status, and changed the global static key
> > > > > > handling so a per-process enable could override the global sysctl.
> > > > > >
> > > > > > This series takes a more conservative variant to preserve existing
> > > > > > administrator and workload expectations. Keeping the global sysctl as a
> > > > > > hard off switch means an administrator can still disable all automatic
> > > > > > NUMA balancing activity with one knob. Using a weak two-state
> > > > > > per-process opt-out/allow ABI avoids exposing a "default" state whose
> > > > > > meaning depends on global policy and is therefore easy to misinterpret.
> > > > > > Storing the configured mode in signal_struct gives the control explicit
> > > > > > thread-group semantics and avoids making unrelated non-thread CLONE_VM
> > > > > > users share the policy merely because they share an mm.
> > > > > >
> > > > > > The series implements a per-process interface: a thread group may use
> > > > > > prctl() to set or get its configured automatic NUMA balancing mode. A
> > > > > > process can opt out of new periodic NUMA scan scheduling and NUMA
> > > > > > scheduler accounting, or opt back in. Existing hinting faults and queued
> > > > > > work are allowed to drain naturally as the disabled state takes effect.
> > > > > > When kernel.numa_balancing is 0, a process-local enable request does not
> > > > > > override it.
> > > > > >
> > > > > > A separate task_struct scheduler snapshot is used by enqueue/dequeue
> > > > > > accounting so the scheduler can consistently account the state that was
> > > > > > in effect when a task entered the runqueue. This avoids changing
> > > > > > historical task accounting while still allowing a process mode change to
> > > > > > take effect through the normal dequeue/update/enqueue path.
> > > > > >
> > > > > > The series also exposes the process NUMA balancing mode in
> > > > > > /proc/<pid>/status and documents the new prctl() ABI and its interaction
> > > > > > with the global sysctl.
> > > > > >
> > > > >
> > > > > Can the global key be extended to tri-state, like the THP settings?
> > > >
> > > > IIUC, a BPF-based THP policy is yet to be developed as prctl() doesn't
> > > > fit some use cases, especially mTHP.
> > > >
> > > > > To keep it backward-compatible, echoing 0 and 1 remains the same effect.
> > > > > The user can also echo always, advise, or never to it.
> > > > > We have a similar proposal here for cache-aware scheduling[1]
> > > >
> > > > We are also experimenting with the LLC-aware scheduler on production
> > > > servers, and we want to enable it per cgroup rather than globally
> > > > there as well.  Given the lessons we have learned from THP, what
> > > > about implementing it as a BPF-based feature from the beginning, so
> > > > that the semantics can still change when new use cases show up?
> > >
> > > Some questions about per-cgroup based cache aware scheduling:
> > >
> > > 1) What should be done if you want to group specific processes in different cgroups?
> > > It may not be desirable to put them together in the same cgroup for other reasons
> > > like IO or memory resource policies.
> >
> > Do you mean a multi-container pod? The policy can be applied at either
> > the pod level or the container level.
> >
> >               Pod            <- policy applied at the pod level
> >           /              \
> >   containerA  containerB   <- or at the container level
> >
> > This is essentially a cgroup hierarchy: the pod is the parent cgroup
> > and each container is a child cgroup, so policies (IO/memory) can be
> > enforced at either level.
> >
>
> My understanding is that Tim was talking about a case where A and B do
> not belong to the same parent (arbitrary hierarchy). Say, process T1, T2:
> T1 belongs to cgroup A, T2 belongs to cgroup B, T1 has threads t1.1, t1.2,
> T2 has threads t2.1, t2.2, and t1.1 and t2.1 want to be aggregated together
> because they have shared data.

Thanks for the clarification.

>
> A more complex scenario is when cgroup A and cgroup B become threaded domains:
>
>                          root
>                         /    \
>                        X      Y
>                        |      |
>                        A      B          A, B: threaded domains (thread roots)
>                       / \    / \         separate memory.max / io.max
>                     A1   A2 B1  B2       threaded cgroups
>                     ^        ^
>                     |        |
>                     +--------+
>                   want to share one cache group
>
> In the above case, we might need to change the deployed cgroup hierarchy.

I think we can enable LLC-aware scheduling for both A1 and A2 — does
that sound right?

>
> Suppose threads in A1 and threads in A2 belong to the same process Pa,
> this reminds me of the cgroup bpf NUMA balancing case: after enabling
> NUMA balancing for A1 but not setting it for A2, what would be the final
> NUMA balancing decision for process Pa, is it enabled or disabled?
> The NUMA balancing scan/invalid seems to be per process, so if we change
> the setting of its threads, the whole process will be impacted. Or maybe
> we should not enable NUMA balancing for threaded cgroups? Anyway I haven't
> digged into your cgroup bpf patch set yet, let me try to look at it.

cgroup NUMAB works the same way: the setting is per cgroup and opt-in.
For Pa in your example, since its threads spread across A1 and A2, you
would enable it on both — enabling it on only one of them is not
meaningful.

>
> > >
> > > 2) What is the inheritance rule for cache grouping of tasks in a cgroup
> > > with child cgroups?
> > > There may be too many tasks in parent to try to fit them all in a LLC when
> > > there are only certain subset of tasks sharing data that should be grouped
> > > together cache wise.
> >
> > Generally, a cgroup contains only a single main process in most use
> > cases. If multiple tasks have to share a cgroup, they are either
> > identical tasks or tasks with different priorities — in which case you
> > only need to focus on the high-priority ones. That's how it works in
> > practice.
> >
>
> IIUC, what Tim mentioned is about the inheritance relationship: if cgroup A
> has cache-aware enabled, will all its child cgroups enable cache-aware
> recursively? (I think the answer is yes, while for NUMA balancing it depends.)

Right. The cgroup NUMA balancing also works that way — it is inherited
recursively as well.

>
> > >
> > > With prctl, we can be precise in grouping only tasks that we truly
> > > want to be together.  If we want to bunch tasks in a cgroup cache wise,
> > > it is easy to write a daemon to put tasks in a cgroup into the same
> > > cache group.
> >
> > I'm not a big fan of the prctl() approach in practice. Deploying a
> > policy that way requires coordinating with other teams, and the policy
> > may well need to be adjusted later — every adjustment would mean
> > asking another team to make changes, which is a painful round-trip.
> > With BPF, on the other hand, all the work stays within the kernel
> > team, and it's completely transparent to users.
> >
>
> I agree that prctl has limitations, but it provides fine-grained control
> per task/thread. Besides, there might also be other users that do not
> deploy cgroups, so in such use cases, per-task settings like prctl or
> sched_setattr() could be another choice. For example, the schedqos tool
> in Google's usecase, and DeepSeek's reinforcement learning infra[1], have
> leveraged prctl to deploy core-scheduling to avoid noisy neighbours. That
> is to say, I don't think the cgroup-based BPF interface and prctl/sched_setattr
> are mutually exclusive, they can be complementary to each other, IMO.

Thanks for the use cases — they're helpful. I'm not opposed to the
prctl() approach.

-- 
Regards
Yafang