Re: [PATCH 0/4] sched/numa: Add per-process automatic NUMA balancing control

[Yafang Shao]

On Fri, Oct 16, 2026 at 3:58 AM Tim Chen <tim.c.chen@xxxxxxxxxxxxxxx> wrote:
>
> On Tue, 2026-10-13 at 17:31 +0800, Yafang Shao wrote:
> > On Tue, Oct 13, 2026 at 3:52 PM Chen Yu <yu.c.chen@xxxxxxxxx> wrote:
> > >
> > > On Tue, Sep 08, 2026 at 08:24:42PM +0800, Li Zhe wrote:
> > > > Automatic NUMA balancing is controlled globally through the
> > > > kernel.numa_balancing sysctl and can be influenced indirectly through
> > > > memory policy. That works well as a system default, but it is too coarse
> > > > for workloads where a launcher wants most processes to use the default
> > > > policy while a selected process opts out because it already manages NUMA
> > > > placement or cannot afford the sampling overhead.
> > > >
> > > > A recent per-cgroup proposal tried to address this by adding a
> > > > cgroup-local enable knob plus a NUMA_BALANCING_CGROUP mode where
> > > > balancing was disabled by default and then enabled for selected cgroups
> > > > [1]. The discussion also noted that automatic NUMA balancing already
> > > > operates at task/process granularity, that the knob is not a resource
> > > > control attribute, and that task/process controls such as sched_setattr()
> > > > or prctl() may be a better fit while cgroups and cpusets continue to
> > > > describe workload grouping and placement domains.
> > > >
> > > > There is also prior art from 2023 prctl() proposals [2][3]. Those
> > > > versions used a per-mm mode with disabled/enabled/default states, added
> > > > NumaB_mode to /proc/<pid>/status, and changed the global static key
> > > > handling so a per-process enable could override the global sysctl.
> > > >
> > > > This series takes a more conservative variant to preserve existing
> > > > administrator and workload expectations. Keeping the global sysctl as a
> > > > hard off switch means an administrator can still disable all automatic
> > > > NUMA balancing activity with one knob. Using a weak two-state
> > > > per-process opt-out/allow ABI avoids exposing a "default" state whose
> > > > meaning depends on global policy and is therefore easy to misinterpret.
> > > > Storing the configured mode in signal_struct gives the control explicit
> > > > thread-group semantics and avoids making unrelated non-thread CLONE_VM
> > > > users share the policy merely because they share an mm.
> > > >
> > > > The series implements a per-process interface: a thread group may use
> > > > prctl() to set or get its configured automatic NUMA balancing mode. A
> > > > process can opt out of new periodic NUMA scan scheduling and NUMA
> > > > scheduler accounting, or opt back in. Existing hinting faults and queued
> > > > work are allowed to drain naturally as the disabled state takes effect.
> > > > When kernel.numa_balancing is 0, a process-local enable request does not
> > > > override it.
> > > >
> > > > A separate task_struct scheduler snapshot is used by enqueue/dequeue
> > > > accounting so the scheduler can consistently account the state that was
> > > > in effect when a task entered the runqueue. This avoids changing
> > > > historical task accounting while still allowing a process mode change to
> > > > take effect through the normal dequeue/update/enqueue path.
> > > >
> > > > The series also exposes the process NUMA balancing mode in
> > > > /proc/<pid>/status and documents the new prctl() ABI and its interaction
> > > > with the global sysctl.
> > > >
> > >
> > > Can the global key be extended to tri-state, like the THP settings?
> >
> > IIUC, a BPF-based THP policy is yet to be developed as prctl() doesn't
> > fit some use cases, especially mTHP.
> >
> > > To keep it backward-compatible, echoing 0 and 1 remains the same effect.
> > > The user can also echo always, advise, or never to it.
> > > We have a similar proposal here for cache-aware scheduling[1]
> >
> > We are also experimenting with the LLC-aware scheduler on production
> > servers, and we want to enable it per cgroup rather than globally
> > there as well.  Given the lessons we have learned from THP, what
> > about implementing it as a BPF-based feature from the beginning, so
> > that the semantics can still change when new use cases show up?
>
> Some questions about per-cgroup based cache aware scheduling:
>
> 1) What should be done if you want to group specific processes in different cgroups?
> It may not be desirable to put them together in the same cgroup for other reasons
> like IO or memory resource policies.

Do you mean a multi-container pod? The policy can be applied at either
the pod level or the container level.

              Pod            <- policy applied at the pod level
          /              \
  containerA  containerB   <- or at the container level

This is essentially a cgroup hierarchy: the pod is the parent cgroup
and each container is a child cgroup, so policies (IO/memory) can be
enforced at either level.

>
> 2) What is the inheritance rule for cache grouping of tasks in a cgroup
> with child cgroups?
> There may be too many tasks in parent to try to fit them all in a LLC when
> there are only certain subset of tasks sharing data that should be grouped
> together cache wise.

Generally, a cgroup contains only a single main process in most use
cases. If multiple tasks have to share a cgroup, they are either
identical tasks or tasks with different priorities — in which case you
only need to focus on the high-priority ones. That's how it works in
practice.

>
> With prctl, we can be precise in grouping only tasks that we truly
> want to be together.  If we want to bunch tasks in a cgroup cache wise,
> it is easy to write a daemon to put tasks in a cgroup into the same
> cache group.

I'm not a big fan of the prctl() approach in practice. Deploying a
policy that way requires coordinating with other teams, and the policy
may well need to be adjusted later — every adjustment would mean
asking another team to make changes, which is a painful round-trip.
With BPF, on the other hand, all the work stays within the kernel
team, and it's completely transparent to users.

-- 
Regards
Yafang