Re: [PATCH net-next] net/smc: introduce autosplit for smc

[Wenjia Zhang]

On 09.07.24 18:05, Guangguan Wang wrote:
> When sending large size data in TCP, the data will be split into
> several segments(packets) to transfer due to MTU config. And in
> the receive side, application can be woken up to recv data every
> packet arrived, the data transmission and data recv copy are
> pipelined.
>
> But for SMC-R, it will transmit as many data as possible in one
> RDMA WRITE and a CDC msg follows the RDMA WRITE, in the receive
> size, the application only be woken up to recv data when all RDMA
> WRITE data and the followed CDC msg arrived. The data transmission
> and data recv copy are sequential.
>
> This patch introduce autosplit for SMC, which can automatic split
> data into several segments and every segment transmitted by one RDMA
> WRITE when sending large size data in SMC. Because of the split, the
> data transmission and data send copy can be pipelined in the send side,
> and the data transmission and data recv copy can be pipelined in the
> receive side. Thus autosplit helps improving latency performance when
> sending large size data. The autosplit also works for SMC-D.
>
> This patch also introduce a sysctl names autosplit_size for configure
> the max size of the split segment, whose default value is 128KiB
> (128KiB perform best in my environment).
>
> The sockperf benchmark shows 17%-28% latency improvement when msgsize
> > = 256KB for SMC-R, 15%-32% latency improvement when msgsize >= 256KB
> for SMC-D with smc-loopback.
>
> Test command:
> sockperf sr --tcp -m 1048575
> sockperf pp --tcp -i <server ip> -m <msgsize> -t 20
>
> Test config:
> sysctl -w net.smc.wmem=524288
> sysctl -w net.smc.rmem=524288
>
> Test results:
> SMC-R
> msgsize   noautosplit    autosplit
> 128KB       55.546 us     55.763 us
> 256KB       83.537 us     69.743 us (17% improve)
> 512KB      138.306 us    100.313 us (28% improve)
> 1MB        273.702 us    197.222 us (28% improve)
>
> SMC-D with smc-loopback
> msgsize   noautosplit    autosplit
> 128KB       14.672 us     14.690 us
> 256KB       28.277 us     23.958 us (15% improve)
> 512KB       63.047 us     45.339 us (28% improve)
> 1MB        129.306 us     87.278 us (32% improve)
>
> Signed-off-by: Guangguan Wang <guangguan.wang@xxxxxxxxxxxxxxxxx>
> ---
>   Documentation/networking/smc-sysctl.rst | 11 +++++++++++
>   include/net/netns/smc.h                 |  1 +
>   net/smc/smc_sysctl.c                    | 12 ++++++++++++
>   net/smc/smc_tx.c                        | 19 ++++++++++++++++++-
>   4 files changed, 42 insertions(+), 1 deletion(-)

Hi Guangguan,

If I remember correctly, the intention to use one RDMA-write for a 
possible large data is to reduce possible many partial stores. Since 
many year has gone, I'm not that sure if it would still be an issue. I 
need some time to check on it.

BTW, I don't really like the idea to use sysctl to set the 
autosplit_size in any value at will. That makes no sense to improve the 
performance.

Thanks,
Wenjia