Re: [RFC PATCH v7 00/29] Hierarchical Constant Bandwidth Server

[Tejun Heo]

Hello, Yuri.

I only looked at the interface and the user-visible behavior, not the
implementation.

On Mon, Nov 09, 2026 at 05:35:46PM +0100, Yuri Andriaccio wrote:
> - Rework cgroup-v2 interface:
>     + Rename cpu.rt.internal to cpu.rt.effective.local

The new name still doesn't follow the cgroup2 naming convention. Every
.effective and .local file is a suffix on an existing file:
cpuset.cpus.effective, cpuset.cpus.exclusive.effective,
memory.events.local, cpu.stat.local. There is no cpu.rt file, so can
we go with cpu.rt.max.effective.local as suggested for v6?

The file prints a "CPU runtime=R period=P" line for each possible CPU.
It does so even when the value is the same on every CPU, and the CPUs
that the budget doesn't cover are listed with 0. 1k CPU machines are
becoming wide-spread as we speak. This isn't a scalable interface.

Switching between "max" and a budget is refused while the cgroup or its
"max" descendants have RT tasks. That's okay as a restriction for now.
Ideally, the switch wouldn't be restricted as long as the budget is
sufficient, but that can be left for the future.

> `cpu` and `cpuset` controllers integration:
...
> first setup the `cpuset` cpus and later setup the HCBS bandwidth. The current
> integration disallows changing allowed cpus while the HCBS bandwidth is
> reserved (thus the admin must first remove the HCBS reservation, change the
> cpuset, and re-request real-time bandwidth).
...
> dl_server override feature:
> Following the `cpuset` integration, we integrated the multi-runtime/multi-CPU
> feature: the admin can request the override of the default runtime/period pair
> for a specific set of CPUs.

This is my main objection. v7 sits in the middle on both axes. Per-CPU
config is allowed but only to a degree: an override must be within the
cpuset, can't be zero and isn't allowed on a "max" cgroup. cpuset is
integrated but not fully: the budget covers the cpuset as it was when
cpu.rt.max was written, and cpuset changes and CPU offlining are
refused while budgets exist.

Can we separate cpu.rt.max from cpuset? I can see the following paths.

1. Make the allocation global. The interface then becomes system-wide,
   and cpuset and CPU hotplug become transparent for the most part.

2. Make the interface per-CPU. The overrides mostly make it so
   already. Config entries are keyed by cpulists only, with nested
   keys like io.max. Reads collapse the CPUs that share a config into
   one cpulist entry:

     0-1,4-5,7 runtime=400000 period=1000000
     2-3,6 runtime=800000 period=1000000

3. If you want simplicity while staying per-CPU, make every cgroup's
   cpu.rt.max allocate on all CPUs.

In any of these, cpuset and cpu.rt.max are orthogonal, and what users
do with intersecting configs is their problem.

On the per-CPU budgets themselves, they don't match cpu.max, and giving
a budget per CPU is a bit odd when the consumption isn't per CPU. The
threads move around. Maybe it can't be avoided, but I'd like to see a
discussion of the rationale behind the design choice and some
exploration of alternatives.

> Hotplug does not cause issues but HCBS reservations will need to be updated to
> make use of the newly enabled core (the cgroups' HCBS specific cpumask is only
> updated when setting the cgroups' bandwidth).

The cpuset coupling creates all sorts of problems. AFAICS, disabling
the cpuset controller above a budgeted cgroup moves it to the
ancestor's cpuset and changes its effective CPUs without going through
the lockout at all. A newly onlined CPU joins cpuset.cpus.effective and
the cgroup's tasks can run there, but the budget doesn't cover it until
cpu.rt.max is rewritten. A partition change in a sibling isn't refused
and can still move CPUs underneath it.

What users configure on the config surface should always match what's
in effect. Interfaces that let internal state deviate from that are
almost always bad ideas and long-term headaches. Hopefully, the lockout
can go away eventually.

> - PI-boosting will cause starvation if cgroups have zero (internal) runtime.
>   There are many solutions, such as bandwidth inheritance or forcing cgroups to
>   not have zero runtime, but this has yet to be investigated. We are open for
>   advice on the matter.

Zero runtime is a problem without PI too. A budgeted cgroup's own
servers get its budget minus what its budgeted children take. When the
children take all of it, the tasks running on the cgroup's own servers
stall. Those are the internal threads of a threaded cgroup and the
tasks in its "max" descendants, and the latter doesn't need threaded
mode. The tasks are still accepted because the attach and setscheduler
checks look at the configured budget, not what's left.

One option is to treat the RT tasks on a cgroup's servers as usage in
admission. Config changes that would leave a cgroup that has RT tasks
with zero runtime would be refused, and attach and setscheduler would
check what's left instead of the configured budget. There may be
better ways.

PI doesn't need to be perfect. Skipping the boost when the target has
no runtime is fine as long as nothing stalls.

Thanks.

--
tejun