Re: [PATCH net-next V6 2/3] net/mlx5e: Avoid copying payload to the skb's linear part

[Amery Hung]

On Thu, May 7, 2026 at 10:54 AM Tariq Toukan <tariqt@xxxxxxxxxx> wrote:
>
> From: Christoph Paasch <cpaasch@xxxxxxxxxx>
>
> mlx5e_skb_from_cqe_mpwrq_nonlinear() copies MLX5E_RX_MAX_HEAD (256)
> bytes from the page-pool to the skb's linear part. Those 256 bytes
> include part of the payload.
>
> When attempting to do GRO in skb_gro_receive, if headlen > data_offset
> (and skb->head_frag is not set), we end up aggregating packets in the
> frag_list.
>
> This is of course not good when we are CPU-limited. Also causes a worse
> skb->len/truesize ratio,...
>
> So, let's avoid copying parts of the payload to the linear part. We use
> eth_get_headlen() to parse the headers and compute the length of the
> protocol headers, which will be used to copy the relevant bits of the
> skb's linear part.
>
> We still allocate MLX5E_RX_MAX_HEAD for the skb so that if the networking
> stack needs to call pskb_may_pull() later on, we don't need to reallocate
> memory.
>
> This gives a nice throughput increase (ARM Neoverse-V2 with CX-7 NIC and
> LRO enabled):
>
> BEFORE:
> =======
> (netserver pinned to core receiving interrupts)
> $ netperf -H 10.221.81.118 -T 80,9 -P 0 -l 60 -- -m 256K -M 256K
>  87380  16384 262144    60.01    32547.82
>
> (netserver pinned to adjacent core receiving interrupts)
> $ netperf -H 10.221.81.118 -T 80,10 -P 0 -l 60 -- -m 256K -M 256K
>  87380  16384 262144    60.00    52531.67
>
> AFTER:
> ======
> (netserver pinned to core receiving interrupts)
> $ netperf -H 10.221.81.118 -T 80,9 -P 0 -l 60 -- -m 256K -M 256K
>  87380  16384 262144    60.00    52896.06
>
> (netserver pinned to adjacent core receiving interrupts)
>  $ netperf -H 10.221.81.118 -T 80,10 -P 0 -l 60 -- -m 256K -M 256K
>  87380  16384 262144    60.00    85094.90
>
> Additional tests across a larger range of parameters w/ and w/o LRO, w/
> and w/o IPv6-encapsulation, different MTUs (1500, 4096, 9000), different
> TCP read/write-sizes as well as UDP benchmarks, all have shown equal or
> better performance with this patch.
>
> Reviewed-by: Eric Dumazet <edumazet@xxxxxxxxxx>
> Reviewed-by: Saeed Mahameed <saeedm@xxxxxxxxxx>
> Signed-off-by: Christoph Paasch <cpaasch@xxxxxxxxxx>
> Signed-off-by: Dragos Tatulea <dtatulea@xxxxxxxxxx>
> Signed-off-by: Tariq Toukan <tariqt@xxxxxxxxxx>
> ---
>  drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> index 75ccf40a7f17..301b33419207 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
> @@ -1976,6 +1976,8 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
>                                         ALIGN(headlen, sizeof(long)),
>                                         rq->buff.map_dir);
>
> +               headlen = eth_get_headlen(rq->netdev, head_addr, headlen);
> +
>                 frag_offset += headlen;
>                 byte_cnt -= headlen;
>                 linear_hr = skb_headroom(skb);
> @@ -2012,9 +2014,13 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
>
>         if (prog) {
>                 u8 nr_frags_free, old_nr_frags = sinfo->nr_frags;
> +               skb_frag_t *frag = &sinfo->frags[0];
>                 u8 new_nr_frags;
>                 u32 len;
>
> +               headlen = eth_get_headlen(rq->netdev, skb_frag_address(frag),
> +                                         skb_frag_size(frag));
> +
>                 if (mlx5e_xdp_handle(rq, prog, mxbuf)) {

Hello,

Am I understanding correctly that the better performance comes with
the assumption that the XDP does not change headers?

headlen is determined before the XDP program runs. If it push/pop
headers, there could be headers in frags or data in the linear region
after __pskb_pull_tail().

>                         if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)) {
>                                 struct mlx5e_frag_page *pfp;
> @@ -2060,8 +2066,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
>                                 pagep->frags++;
>                         while (++pagep < frag_page);
>
> -                       headlen = min_t(u16, MLX5E_RX_MAX_HEAD - len,
> -                                       skb->data_len);
> +                       headlen = min_t(u16, headlen - len, skb->data_len);

headlen - len can underflow but will be capped by skb->data_len, so
this should be okay, right?

>                         __pskb_pull_tail(skb, headlen);
>                 }
>         } else {
> --
> 2.44.0
>