Re: [PATCH v5 13/29] sched/deadline: Keep departing bandwidth across a root domain rebuild

[Mathieu Desnoyers]

On 2026-11-11 07:59, Juri Lelli wrote:
Hi Juri,

> It took me a bit of staring to (hopefully) understand better how
> dl_bw_gen works, maybe the following can help others?

It does, thanks. v6 takes your comment as is. I only changed the name
of the function that increments the count: it is
dl_clear_root_domains().

> Maybe additional info can again help (before the above)?

Taken as well.

> Can something like the following happen? And do we actually have a test
> for it?

Yes, it happens at this patch, and there was no test.

dl_task_undepart() forgets the CPU recorded. The end of the boost then
takes X from rd(B). In my run the deboost armed the timer again, and
the timer took X from rd(B) at the 0-lag time. total_bw of rd(B) went
negative, and rd(A) kept X.

The end of the series hides it. Patch 16 replaces dl_departed_cpu with
dl_bw_cpu, and the record survives the boost. So only a kernel stopped
between patches 13 and 15 shows it.

Planned for v6:
- This patch keeps the CPU recorded when a boost takes the task off the
  list. The series then bisects.
- deadline_bw_test gets a case for your scenario, pi_departed_moved.
  It fails on v5 at this patch, and passes on v6 at this patch and at
  the end.

> Wouldn't it be better to move to a single list of accounted tasks
> (current and departed alike), keyed by their recorded home CPU, which
> turns the rebuild from a reconstruction into a re-partition? The
> dl_bw_gen machinery and the cpuset walk would both go away.

I tried it. It works, and the code gets simpler. But it costs
interrupt latency, so I left it out of v6.

What I tried:
- Every task whose bandwidth is accounted is on one list. It joins at
  admission and leaves at the release. A departed task is marked.
- A rebuild clears the root domains and counts the list again under one
  lock. A release or a move sees the accounting before or after it.
- dl_bw_gen goes away, with the walk of the task list (which replaced
  the cpuset walk in patch 15) and dl_add_task_root_domain().

The 75 selftest cases pass with it. One point differs from your
description. The rebuild is not a pure re-partition by the recorded
CPU. A cpuset change sets the masks of the tasks, then rebuilds. A
sleeping task must then be counted where its new mask puts it. So the
rebuild still picks the CPU of a deadline task from its mask, as patch
22 does.

The problem is the lock. A release takes it under the rq lock, so it is
a raw spinlock taken with interrupts disabled. The rebuild has to hold
it from the clear to the last task. Otherwise a release sees a
half-rebuilt accounting, and needs dl_bw_gen again.

Measured in a 64-vCPU guest with sleeping deadline tasks:

  tasks    today: rebuild, longest irqs-off    single list, all irqs-off
  1000     0.75ms, under 0.1ms                 0.12ms (0.19ms worst)
  10000    7.4ms, under 0.1ms (0.3ms once)     0.9ms (1.2ms worst)

The rebuild itself is 8 times shorter. That comes from updating the
extra_bw of the CPUs once per root domain, not once per task. But the
longest section with interrupts disabled grows with the number of
deadline tasks. I do not think that is a step forward for a real-time
class.

The patch is on a branch. I can post it as an RFC after this series if
you want to look at it. A walk that drops the lock every N tasks would
bound the latency, but it needs a per-task marker again.

Thanks,

Mathieu


--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com