RE: [PATCH net-next v3] net: mana: add BQL support

[Haiyang Zhang]

> -----Original Message-----
> From: Hamza Mahfooz <hamzamahfooz@xxxxxxxxxxxxxxxxxxx>
> Sent: Tuesday, October 13, 2026 5:50 PM
> To: netdev@xxxxxxxxxxxxxxx
> Cc: Haiyang Zhang <haiyangz@xxxxxxxxxxxxx>; Wei Liu <wei.liu@xxxxxxxxxx>;
> Dexuan Cui <DECUI@xxxxxxxxxxxxx>; Andrew Lunn <andrew+netdev@xxxxxxx>;
> David S. Miller <davem@xxxxxxxxxxxxx>; Eric Dumazet <edumazet@xxxxxxxxxx>;
> Jakub Kicinski <kuba@xxxxxxxxxx>; Paolo Abeni <pabeni@xxxxxxxxxx>;
> Konstantin Taranov <kotaranov@xxxxxxxxxxxxx>; Simon Horman
> <horms@xxxxxxxxxx>; Alexei Starovoitov <ast@xxxxxxxxxx>; Daniel Borkmann
> <daniel@xxxxxxxxxxxxx>; Jesper Dangaard Brouer <hawk@xxxxxxxxxx>; John
> Fastabend <john.fastabend@xxxxxxxxx>; Stanislav Fomichev
> <sdf@xxxxxxxxxxx>; linux-hyperv@xxxxxxxxxxxxxxx; linux-
> kernel@xxxxxxxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; bpf@xxxxxxxxxxxxxxx;
> Hamza Mahfooz <hamzamahfooz@xxxxxxxxxxxxxxxxxxx>
> Subject: [PATCH net-next v3] net: mana: add BQL support
> 
> MANA doesn't support byte queue limits, so each TX queue can hold 256
> WQEs by default, and up to 16384, of up to 64 KB TSO packets, before the
> qdisc layer sees any backpressure. This adds latency to all the flows
> sharing a queue, and defeats AQM qdiscs such as fq_codel, which rely on
> BQL to keep the driver queues short.
> 
> Report the bytes queued and completed to BQL, except for the XDP_TX and
> XDP_REDIRECT frames. These share the SQs, but bypass the qdisc and are
> sent even when the queue is stopped, so BQL can't hold them back.
> Counting them would let sustained XDP traffic keep the queue stopped,
> starving the stack, and the TX watchdog would then either fire while the
> completions still arrive, or, if the XDP sends refreshed trans_start,
> not fire when the completions stop. The XDP SKBs are marked in their
> skb->cb, next to the WQE size, so that their completions are skipped as
> well.
> 
> Reset the BQL state when destroying the queues, once their NAPI is
> disabled, since the SKBs still pending after a drain timeout are freed
> without being completed.
> 
> Lost TX completions can now stop a queue sooner: it also stops when the
> bytes in flight exceed the BQL limit, not only when the SQ is full, and
> the limit may never grow enough to cover the lost bytes. This includes
> the completions skipped by the WARN_ON_ONCE() early returns in
> mana_poll_tx_cq(), which already leak SQ space: completing the SKBs
> already freed there wouldn't reliably balance BQL, since
> mana_gd_poll_cq() has already consumed the CQEs of the rest of the
> batch. The TX watchdog then fires, as nothing refreshes the trans_start
> of a queue stopped by BQL, and its handler resets the queues along with
> their BQL state. The exception, as before this patch, is XDP traffic
> keeping the SQ full: each XDP send then stops the queue again,
> refreshing its trans_start.
> 
> Tested on two Standard_D16ds_v6 Azure VMs in a proximity placement
> group, where the throughput is capped at 10.6 Gbps, with the DUT
> alternately booting net-next (c1c1f0a31712) with and without this patch
> applied.
> 
> The throughput stays at the cap with 1, 8 and 64 TCP flows, and the idle
> TCP_RR latency doesn't change (p50 of 47 us with vs. without the patch).
> With 16 senders of 64-byte UDP packets, the DUT sends 3.7 to 4.0 Mpps
> without the patch and 3.7 to 4.1 Mpps with it. With bulk TCP flows
> sharing the queues, the netperf TCP_RR latency drops (medians of 6
> runs):
> 
>   16 queues, 64 flows: p50 4649 -> 2182 us, p99 5621 -> 2677 us
>   1 queue, 8 flows:    p50 2554 ->  418 us, p99 2723 ->  624 us
> 
> The standing queue moves from the SQs to fq_codel, which now drops
> packets: about 2500 qdisc drops and 88K TCP retransmissions per 48 s run
> with 64 flows, against at most 29 retransmissions before. The
> retransmissions are about 0.2% of the segments sent. The TCP_RR flows
> lose no packets, and the packets of the bulk flows are dropped before
> they reach the wire, so the goodput changes by less than 0.01%.
> 
> Signed-off-by: Hamza Mahfooz <hamzamahfooz@xxxxxxxxxxxxxxxxxxx>

Thank you!
Reviewed-by: Haiyang Zhang <haiyangz@xxxxxxxxxxxxx>