Re: [PATCH net-next] net/sched: sch_cake: skip clearing unused tins during rate adjustment

[Jonas Köppeler]

On 7/17/26 10:31, Toke Høiland-Jørgensen wrote:
> Jonas Köppeler <j.koeppeler@xxxxxxxxxxxx> writes:
>
> > When cake_configure_rates() is called from the dequeue path with
> > rate_adjust=true, it only needs to update the rate parameters. The
> > loop that clears the unused tins is both unnecessary and harmful in
> > this path:
> >
> >   - cake_clear_tin() overwrites q->cur_tin and q->cur_flow, which are
> >     actively used by cake_dequeue(), corrupting the dequeue state.
> >   - iterating over the unused tins and their internal queues to purge
> >     packets adds needless overhead to the hot path.
> >
> > Skip the entire loop when rate_adjust is set, as neither
> > cake_clear_tin() nor the mtu_time update are needed when only the
> > rate changes.
> >
> > Fixes: 15c2715a5264 ("net/sched: sch_cake: fixup cake_mq rate adjustment for diffserv config")
> > Signed-off-by: Jonas Köppeler <j.koeppeler@xxxxxxxxxxxx>
> > Tested-by: Mike Pham <mikepham4321@xxxxxxxxx>
>
> Do you have any performance numbers to show the impact of this?
Yes, the table below shows results from a test setup using vng with
2 network namespaces, with cake/cake_mq attached in one of them:

    ns1 -> cake/cake_mq -> ns2

- veth devices are configured with 8 rx/tx queues.
- cake/cake_mq is configured with a 2 Gbit rate limit.
- Running flent's rrul and tcp_nup tests with 32 TCP upstreams:

legend: qdisc mq = cake_mq; mode be = besteffort, ds3 = diffserv3
        test nup = tcp_nup; base/load = idle/loaded RTT (ms); tput = Mbit/s

+---------------------+-------+------+------+-------+-------+---------+
| kernel              | qdisc | mode | test |  base |  load |    tput |
+---------------------+-------+------+------+-------+-------+---------+
| net-next            | cake  | be   | rrul | 0.075 |  4.76 | 1473.69 |
| net-next            | cake  | be   | nup  | 0.078 |  6.23 | 1550.79 |
| net-next            | cake  | ds3  | rrul | 0.063 |  5.81 | 1526.75 |
| net-next            | cake  | ds3  | nup  | 0.046 |  6.09 | 1761.45 |
+---------------------+-------+------+------+-------+-------+---------+
| net-next            | mq    | be   | rrul | 0.810 | 11.78 | 1469.67 |
| net-next            | mq    | be   | nup  | 0.637 | 85.71 | 1243.15 |
| net-next            | mq    | ds3  | rrul | 0.397 | 15.28 | 1770.06 |
| net-next            | mq    | ds3  | nup  | 0.351 | 15.98 | 1799.39 |
+---------------------+-------+------+------+-------+-------+---------+
| this patch          | mq    | be   | rrul | 0.092 |  0.56 | 1873.40 |
| this patch          | mq    | be   | nup  | 0.109 |  1.82 | 1869.12 |
| this patch          | mq    | ds3  | rrul | 0.097 |  0.98 | 1866.10 |
| this patch          | mq    | ds3  | nup  | 0.101 |  0.51 | 1861.79 |
+---------------------+-------+------+------+-------+-------+---------+
| before 15c2715a5264 | mq    | be   | rrul | 0.073 |  0.30 | 1895.45 |
| before 15c2715a5264 | mq    | be   | nup  | 0.076 |  0.49 | 1905.57 |
| before 15c2715a5264 | mq    | ds3  | rrul | 0.069 |  0.31 | 1896.59 |
| before 15c2715a5264 | mq    | ds3  | nup  | 0.058 |  0.86 | 1884.01 |
+---------------------+-------+------+------+-------+-------+---------+

Not only is p99 latency drastically reduced -- nearly matching
pre-15c2715a5264 results -- but on current upstream cake_mq,
throughput also increases as a cake mode uses more tins. This points
directly to cake_clear_tin() during reconfig as the cause, since it
clears (max_tins - cur_tins) tins each time. So the fewer tins the
current mode uses, the more get cleared on every reconfig.

Mike ran also some test on OpenWrt, on an IPQ8074A with 4 rx/tx
queues, and saw similar trends. cake_mq is configured with a 2.2 Gbit
rate limit.

Unfortunately, we only have data for 128 TCP upstreams on net-next,
and 64 TCP upstreams for 'this patch'.

+---------------------+-------+------+------+---------+----------+
| kernel              | qdisc | mode | test |    load |     tput |
+---------------------+-------+------+------+---------+----------+
| net-next            | mq    | be   | nup  |  468.50 |    50.90 |
| net-next            | mq    | ds3  | nup  |  355.22 |    98.21 |
| net-next            | mq    | ds4  | nup  |  268.28 |   255.84 |
| net-next            | mq    | ds8  | nup  |    7.48 |  2023.66 |
+---------------------+-------+------+------+---------+----------+
| this patch          | mq    | be   | nup  |    4.24 |   944.35 |
| this patch          | mq    | ds3  | nup  |    4.27 |   937.75 |
| this patch          | mq    | ds4  | nup  |    4.24 |   936.97 |
| this patch          | mq    | ds8  | nup  |    4.32 |   927.89 |
+---------------------+-------+------+------+---------+----------+

This again shows the same trend: throughput increases and latency
drops as cake_mq is configured with more tins. We're still looking
into why net-next+ds8 reaches close to 2 Gbit/s, while this patch
tops out around 928 Mbit/s.

That said, this patch doesn't solve every issue yet, but it does
remove the regression introduced by commit 15c2715a5264
("net/sched: sch_cake: fixup cake_mq rate adjustment for diffserv
config").

We're continuing to look into further improvements. Let us know if
you'd like to see additional tests :)

- Jonas

> -Toke