Re: [PATCH] dl_server: Reset DL server params when rd changes

[Juri Lelli]

Hi Joel,

On 29/10/24 22:51, Joel Fernandes (Google) wrote:
> During boot initialization, DL server parameters are initialized using the
> default root domain before the proper scheduler domains and root domains
> are built. This results in DL server parameters being tied to the default
> root domain's bandwidth accounting instead of the actual root domain
> assigned to the CPU after scheduler topology initialization.
> 
> When secondary CPUs are brought up, the dl_bw_cpus() accounting doesn't
> properly track CPUs being added since the DL server was started too early
> with the default root domain. Specifically, dl_bw_cpus() is called before
> set_cpu_active() during secondary CPU bringup, causing it to not account
> for the CPU being brought up in its capacity calculations. This causes
> subsequent sysfs parameter updates to fail with -EBUSY due to bandwidth
> accounting using the wrong root domain with zeroed total_bw.
> 
> This issue also causes under-utilization of system capacity. With the fix,
> we see proper capacity initialization and scaling as CPUs come online - the
> total system capacity increases from CPU 0 to CPU 1 and continues scaling
> up as more CPUs are added (from cap=1024 initially to cap=8192 with 8
> CPUs). Without the fix, the capacity initialization was incomplete since
> dl_bw_cpus() runs before the CPU is marked active in set_cpu_active(),
> leading to CPUs not being properly accounted for in the capacity
> calculations.
> 
> Fix this by tracking the last root domain used for the DL server and
> resetting the server parameters when the root domain changes. This ensures
> bandwidth accounting uses the correct, fully initialized root domain after
> the scheduler topology is built.

So, I'm trying to reproduce this issue, but currenlty not really seeing
it, sorry.

I'm on a 40 CPUs box and, even if I fiddle with hotplug, the numbers I
see from debug (bw, total_bw) seem sane and consistent with the fair
server settings.

Could you please provide additional info about how you reproduce the
issue? Maybe you have a test script around you could share?

Thanks!
Juri