[REGRESSION 6.12 -> 6.18] fair (dl-)server delivers no bandwidth to CFS under a CPU-bound RT task on a nohz_full isolated core

[Ionut Nechita (Wind River)]

Hi,

I'm chasing what looks like a fair (dl-)server regression between 6.12 and
6.18 and wanted to ask whether this behaviour is known/expected before I
spend build cycles on a full bisect.

Question in one line: on a nohz_full isolated core, with a CPU-bound
SCHED_RR task pinned to it, should the fair_server still guarantee its
50ms/1s (5%) to a co-scheduled SCHED_OTHER task? On 6.12 it does; on 6.18
it delivers essentially nothing (~0.2%).

## Environment

  - Intel Xeon Gold 6338N @ 2.20 GHz, Ice Lake-SP (family 6, model 106)
    64 logical CPUs (32 cores, 2 threads/core, 1 socket, 1 NUMA node)
    HT enabled; test CPU (cpu5) is in the isolated set
  - Debian-packaged kernels, CONFIG_PREEMPT_RT=y, CONFIG_PREEMPT_LAZY=y
  - Boot cmdline (relevant parts):
      nohz_full=2-28,34-60
      isolcpus=nohz,domain,managed_irq,2-28,34-60
      rcu_nocbs=2-31,34-63
      kthread_cpus=0-1,32-33
      irqaffinity=29-31,61-63
      skew_tick=1 rcutree.kthread_prio=21
      intel_pstate=none nopti nospectre_v2 nospectre_v1
  - CONFIG_RT_GROUP_SCHED is NOT set
  - CONFIG_FAIR_GROUP_SCHED=y, CONFIG_CFS_BANDWIDTH=y, HZ=1000
  - kernel.sched_rt_runtime_us = -1  (RT throttling disabled;
    consequently dl_rq: dl_bw->bw = -1)
  - fair_server: runtime=50000000 period=1000000000 (defaults)

The two kernels compared use an otherwise identical scheduler config
(RT_GROUP_SCHED off, NO_HZ_FULL, CFS_BANDWIDTH, HZ, preempt model). The
only sched-relevant Kconfig delta is CONFIG_GROUP_SCHED_BANDWIDTH=y on
6.18 (absent on 6.12).

## Reproducer

On an isolated CPU $C, run a CFS hog and an RT hog pinned to the same CPU,
then measure how much on-CPU time the CFS task accrues over 6s via
/proc/PID/schedstat field 1:

  taskset -c $C cat /dev/zero >/dev/null & CFS=$!
  chrt -r 25 taskset -c $C cat /dev/zero >/dev/null & RT=$!
  ( sleep 10; kill -9 $CFS $RT 2>/dev/null ) &
  sleep 6
  awk '{printf "CFS ms/s = %d\n", $1/6000000}' /proc/$CFS/schedstat
  kill -9 $CFS $RT 2>/dev/null

## Results (same config, same test)

  Kernel                     CFS bandwidth delivered   local-timer ticks / 6s
  -------------------------  ------------------------  -----------------------
  6.12.57-rt (good)          ~50 ms/s  (5%, expected)  ~360  (tick mostly off)
  6.18.15-rt (bad)           ~2  ms/s  (0.2%)          ~10000 (tick kept on)

So on 6.12 the fair_server delivers the full 5% to CFS even though the
tick on the isolated CPU is (correctly) mostly stopped. On 6.18 the CFS
task is effectively starved by the RR task, even though the tick is
running at HZ on that CPU.

dl_rq snapshot on the isolated CPU under load (same test, both kernels):

  6.12.57-rt (good):  dl_nr_running = 0 ; dl_bw->bw = -1 ; dl_bw->total_bw = 0
  6.18.15-rt (bad):   dl_nr_running = 0 ; dl_bw->bw = -1 ; dl_bw->total_bw = 2831112

The interesting delta is dl_bw->total_bw: it is 0 on 6.12 but non-zero on
6.18 (looks like the sum of the per-CPU fair_server reservations). On 6.18
the fair_server bandwidth is reserved in dl_bw yet the deferred server
never actually runs; on 6.12 nothing is reserved and CFS gets its share.
This makes me suspect the dl_server bandwidth accounting/reservation change
(combined with dl_bw->bw = -1 from disabled RT throttling) as the trigger.

## What I suspect (not yet bisected)

I have not been able to bisect yet. Scanning v6.12..v6.18 for dl_server
changes, the one whose intent matches the symptom ("server delivers less")
is:

  cccb45d7c4295 ("sched/deadline: Less agressive dl_server handling")
                v6.17-rc1

followed by a train of follow-up fixes in the same area:

  fc975cfb36393 v6.16-rc5  Fix dl_server runtime calculation formula
  421fc59cf58c6 v6.17-rc4  Fix RT task potential starvation when expiry ...
  4717432dfd99b v6.17-rc4  Fix dl_server_stopped()
  4ae8d9aa9f9dc v6.17      Fix dl_server getting stuck
  a3a70caf79067 v6.17      Fix dl_server behaviour
  36370892e3111 v6.18.9    Fix 'stuck' dl_server
  909131ad0a006 v6.18.14   Clear the defer params

Note 909131ad0a006 is already present in the 6.18.15 I tested, and the
starvation persists, so whatever changed is not addressed by the latest
defer-params fix in my tree.

## Questions

  1. Is CFS getting ~0% (rather than the 5% fair_server guarantee) under a
     same-core CPU-bound RR task on a nohz_full/isolated CPU a known
     regression, or is it now considered expected with the deferred
     server + sched_rt_runtime_us=-1?
  2. Is the deferred fair_server expected to run at all when RT throttling
     is disabled (dl_bw = -1)?
  3. Is cccb45d7c4295 a plausible first-bad candidate here, or should I
     look elsewhere in the 6.17 window?

I'm happy to run a full bisect and/or targeted traces (sched_switch,
dl_server events) if that helps narrow it down.

Thanks,
Ionut Nechita