Re: [PATCH net-next] net/smc: introduce autosplit for smc

[Guangguan Wang]

On 2024/7/15 10:53, Guangguan Wang wrote:
> 
> 
> On 2024/7/11 23:57, Wenjia Zhang wrote:
>>
>>
>> On 09.07.24 18:05, Guangguan Wang wrote:
>>> When sending large size data in TCP, the data will be split into
>>> several segments(packets) to transfer due to MTU config. And in
>>> the receive side, application can be woken up to recv data every
>>> packet arrived, the data transmission and data recv copy are
>>> pipelined.
>>>
>>> But for SMC-R, it will transmit as many data as possible in one
>>> RDMA WRITE and a CDC msg follows the RDMA WRITE, in the receive
>>> size, the application only be woken up to recv data when all RDMA
>>> WRITE data and the followed CDC msg arrived. The data transmission
>>> and data recv copy are sequential.
>>>
>>> This patch introduce autosplit for SMC, which can automatic split
>>> data into several segments and every segment transmitted by one RDMA
>>> WRITE when sending large size data in SMC. Because of the split, the
>>> data transmission and data send copy can be pipelined in the send side,
>>> and the data transmission and data recv copy can be pipelined in the
>>> receive side. Thus autosplit helps improving latency performance when
>>> sending large size data. The autosplit also works for SMC-D.
>>>
>>> This patch also introduce a sysctl names autosplit_size for configure
>>> the max size of the split segment, whose default value is 128KiB
>>> (128KiB perform best in my environment).
>>>
>>> The sockperf benchmark shows 17%-28% latency improvement when msgsize
>>>> = 256KB for SMC-R, 15%-32% latency improvement when msgsize >= 256KB
>>> for SMC-D with smc-loopback.
>>>
>>> Test command:
>>> sockperf sr --tcp -m 1048575
>>> sockperf pp --tcp -i <server ip> -m <msgsize> -t 20
>>>
>>> Test config:
>>> sysctl -w net.smc.wmem=524288
>>> sysctl -w net.smc.rmem=524288
>>>
>>> Test results:
>>> SMC-R
>>> msgsize   noautosplit    autosplit
>>> 128KB       55.546 us     55.763 us
>>> 256KB       83.537 us     69.743 us (17% improve)
>>> 512KB      138.306 us    100.313 us (28% improve)
>>> 1MB        273.702 us    197.222 us (28% improve)
>>>
>>> SMC-D with smc-loopback
>>> msgsize   noautosplit    autosplit
>>> 128KB       14.672 us     14.690 us
>>> 256KB       28.277 us     23.958 us (15% improve)
>>> 512KB       63.047 us     45.339 us (28% improve)
>>> 1MB        129.306 us     87.278 us (32% improve)
>>>
>>> Signed-off-by: Guangguan Wang <guangguan.wang@xxxxxxxxxxxxxxxxx>
>>> ---
>>>   Documentation/networking/smc-sysctl.rst | 11 +++++++++++
>>>   include/net/netns/smc.h                 |  1 +
>>>   net/smc/smc_sysctl.c                    | 12 ++++++++++++
>>>   net/smc/smc_tx.c                        | 19 ++++++++++++++++++-
>>>   4 files changed, 42 insertions(+), 1 deletion(-)
>>>
>>
>> Hi Guangguan,
>>
>> If I remember correctly, the intention to use one RDMA-write for a possible large data is to reduce possible many partial stores. Since many year has gone, I'm not that sure if it would still be an issue. I need some time to check on it.
>>
> 
> Did you mean too many partial stores will result in some issue? What's the issue?
> 
> 
>> BTW, I don't really like the idea to use sysctl to set the autosplit_size in any value at will. That makes no sense to improve the performance.
> 
> Although 128KB autosplit_size have a good performance in most scenario, I still found some better autosplit_size for some specific network configurations.
> For example, 128KB autosplit_size have a good performance whether the MTU is 1500 or 8500, but for 8500 MTU, 64KB autosplit_size performs better.
> 
> Maybe the sysctl is not the best way, but I think it should have a way to set the value of autosplit_size for possible performance tuning.
> 
> Thanks,
> Guangguan Wang
> 

Hi Wenjia,

Is there any update comment or information about this patch?

>>
>> Thanks,
>> Wenjia