Re: [PATCH RFC v3] sched/proxy: Defer donor commit until after proxy resolution

[Xukai Wang]

Hi Prateek, John, all,

Quick ping on the v3 defer-donor-commit patch. Following up on John's
questions from the v2 review -- specifically what the real impact of this
change is and whether it's measurable -- I've collected targeted
measurements of the put_prev_set_next_task() overhead on the donor commit
path.

1. Test setup

I ran John's priority-inversion-demo workload [1] 3 times per kernel on a
bare-metal x86 system.

I instrumented the two scheduling-class callbacks invoked by
put_prev_set_next_task() on the donor commit path:

/* around put_prev_task() */
t0 = local_clock();
prev->sched_class->put_prev_task(...);
t1 = local_clock();

put_calls++;
put_total_ns += t1 - t0;

/* around set_next_task() */
t0 = local_clock();
next->sched_class->set_next_task(...);
t1 = local_clock();

set_calls++;
set_total_ns += t1 - t0;

I also calibrated the overhead of an empty local_clock() pair (minimum
15ns observed on this machine) and subtracted that per-call from the
measured totals.

t0 = local_clock();
t1 = local_clock();

The adjusted times below subtract that value once per measured callback:

adjusted_total = raw_total - nr_calls * 15 ns

2. Results
2.1 callback invocations

Combined across 3 runs:

                  Baseline    v3 patch
  put_prev_task   49037       8968
  set_next_task   49037       8968

That is an 81.7% reduction in scheduling-class callback invocations
from the donor commit path.


2.2 callback execution time

Adjusted total time spent in these callbacks per workload run:

            Baseline    v3 patch
  Run 1     11.131 ms   2.361 ms
  Run 2     11.157 ms   2.548 ms
  Run 3     11.578 ms   2.642 ms
  Average   ~11.29 ms   ~2.52 ms

For reference, a single idle -> fair transition (the common case in these
runs) costs ~0.69 µs on baseline:
  put_prev_task(idle):  142-154 ns
  set_next_task(fair):  503-557 ns

3.End-to-End latency

I also ran uninstrumented kernels to check full workload end-to-end
impact. I did not observe a consistent stable shift:
  p99  latency: ~5.00 ms on both baseline and v3
  p999 latency: ~9.00 ms on both baseline and v3

The reduction above amounts to about 8.8 ms of scheduling-class callback
time per workload run. This local reduction did not translate into a
stable end-to-end latency shift in this workload.

So the measurable effect is at the donor commit point itself: deferring
the commit reduced the number of put_prev_task()/set_next_task() callback
invocations by 81.7%, with the measured callback time at that point
dropping from about 11.29 ms to 2.52 ms per workload run.

Separately from the measured cost, this also places the scheduling-class
state commit after the donor has survived proxy resolution, rather than
committing state for a candidate that may subsequently be abandoned.

I'd appreciate your thoughts on whether these measurements address the
concern from v2, and whether there are any changes you'd like to see for
v4.

LINK: https://github.com/johnstultz-work/priority-inversion-demo [1]