Re: [RFC PATCH v3 bpf-next 0/5] BPF-based per-cgroup NUMA balancing mode selection

[Yafang Shao]

On Fri, Oct 23, 2026 at 2:32 PM Li Zhe <lizhe.67@xxxxxxxxxxxxx> wrote:
>
> On 10/19/26 2:41 PM, Yafang Shao wrote:
> > The Problem
> > ===========
> >
> > In our large fleet of Kubernetes-managed servers, NUMA balancing has
> > been historically disabled globally on each server. With the increasing
> > deployment of multi-socket AMD EPYC servers, cross-NUMA access has
> > become a critical performance issue, which made us reconsider enabling NUMA
> > balancing to address it.
> >
> > Enabling NUMA balancing globally is still not acceptable since it can
> > introduce latency spikes for latency-sensitive workloads. What we need is
> > to choose per workload whether NUMA balancing runs at all, and which mode
> > it runs in. A latency-sensitive service wants NUMA balancing disabled,
> > while a batch service wants its PMEM-resident memory promoted when hot
> > without paying for scheduler-driven migration. No single global setting
> > serves both.
> >
> > The Design
> > ==========
> >
> > This series introduces the fine-grained per-cgroup NUMA balancing
> > control. It grows out of Tejun's suggestion to store the existing
> > NUMA_BALANCING_* bitmask per cgroup and expose it through BPF kfuncs.
> > cgroup::numab_mode holds the request: NUMAB_FOLLOW_SYSTEM (-1) defers
> > to the global sysctl state, anything else is a bitmask.
> > bpf_numab_set_mode() writes a request, bpf_numab_get_mode() reads the
> > combined request of the cgroup and of every ancestor back. The hot
> > paths walk the same chain at read time behind a static key, so while no
> > cgroup makes a request the overhead is one patched-out branch.
> >
> > An opt-out wins and binds the subtree: a DISABLED request anywhere in
> > the chain opts the cgroup's tasks out entirely. Otherwise the requested
> > bits OR together and the kernel reads them independently, the way
> > kernel.numa_balancing reads them, so a combined NORMAL|TIERING engages
> > both modes. A DISABLED request only speaks for the cgroup's tasks,
> > never for its memory, which follows the remaining requests of its
> > ancestors.
> >
> > The cost of the additive combination is that an ancestor cannot cap
> > tiering short of a full opt-out: a descendant's TIERING request
> > re-engages promotion for its own memory, and promotion is the mode that
> > consumes the shared fast tier.
>
>
> I am not sure that this matches the usual cgroup hierarchy semantics.

I think the usual cgroup hierarchy semantics being invoked here is
the semantics of resource limits: an ancestor's limit binds every
descendant.  That is the right semantics for limits: a limit
partitions a shared, finite resource, and what a descendant
consumes comes out of its ancestors' pool, so an ancestor's limit
must bind every level below it.

A policy request is a different kind of knob: it is a preference
about how the kernel should treat the workload, not a quantity
drawn from a parent's pool.  Nothing is consumed by expressing it,
so the limit semantics does not carry over; the question is only
how the preferences of several levels should combine.

We chose the additive answer: a descendant may ask for more on top
of what its ancestors asked for, while an opt-out at any level
still wins - DISABLED anywhere in the tree caps the whole
subtree.

This also matches our operational experience with policy-type
knobs in our Kubernetes fleet: letting a child refine the result
has worked better than having the parent's choice bind the child.

So with
                      Pod     (NORMAL)
                  /              \
  cgroup-A (none)  cgroup-B (TIERING)

the effective modes are

                           Pod     (NORMAL)
                  /                      \
  cgroup-A (NORMAL)  cgroup-B (TIERING|NORMAL)

> If promotion consumes the shared fast tier, why is a descendant allowed
> to re-enable TIERING when an ancestor cannot cap it short of a full
> opt-out?

Promotion can't make a cgroup consume more memory than it may
already consume: it moves the cgroup's own folios, within its own
memory limit, from the slow tier into the fast tier.  The cost to
the rest of the system is placement contention on the fast tier,
which each contributor's own limit bounds.

What an ancestor can't do now is cap only tiering while leaving
normal balancing available below.  If that targeted cap is wanted,
the natural extension is a negative request: NO_TIERING would
propagate down the tree like the positive requests and mask
TIERING out of the effective mode.  With

                  Pod (NORMAL | NO_TIERING)
                     /            \
            cgroup-A (none)   cgroup-B (TIERING)

the effective modes would be

                      Pod (NORMAL)
                     /            \
            cgroup-A (NORMAL)  cgroup-B (NORMAL)

For the RFC I would keep the modes minimal and add NO_TIERING
as a follow-up once there is a real user for it.  If you think the
targeted cap should be there from the start, it is a small
addition.  What do you think?

--
Regards
Yafang