Re: [PATCH net-next] net/smc: introduce autosplit for smc
From: Guangguan Wang
Date: Thu Aug 08 2024 - 02:27:20 EST
On 2024/7/15 10:53, Guangguan Wang wrote:
>
>
> On 2024/7/11 23:57, Wenjia Zhang wrote:
>>
>>
>> On 09.07.24 18:05, Guangguan Wang wrote:
>>> When sending large size data in TCP, the data will be split into
>>> several segments(packets) to transfer due to MTU config. And in
>>> the receive side, application can be woken up to recv data every
>>> packet arrived, the data transmission and data recv copy are
>>> pipelined.
>>>
>>> But for SMC-R, it will transmit as many data as possible in one
>>> RDMA WRITE and a CDC msg follows the RDMA WRITE, in the receive
>>> size, the application only be woken up to recv data when all RDMA
>>> WRITE data and the followed CDC msg arrived. The data transmission
>>> and data recv copy are sequential.
>>>
>>> This patch introduce autosplit for SMC, which can automatic split
>>> data into several segments and every segment transmitted by one RDMA
>>> WRITE when sending large size data in SMC. Because of the split, the
>>> data transmission and data send copy can be pipelined in the send side,
>>> and the data transmission and data recv copy can be pipelined in the
>>> receive side. Thus autosplit helps improving latency performance when
>>> sending large size data. The autosplit also works for SMC-D.
>>>
>>> This patch also introduce a sysctl names autosplit_size for configure
>>> the max size of the split segment, whose default value is 128KiB
>>> (128KiB perform best in my environment).
>>>
>>> The sockperf benchmark shows 17%-28% latency improvement when msgsize
>>>> = 256KB for SMC-R, 15%-32% latency improvement when msgsize >= 256KB
>>> for SMC-D with smc-loopback.
>>>
>>> Test command:
>>> sockperf sr --tcp -m 1048575
>>> sockperf pp --tcp -i <server ip> -m <msgsize> -t 20
>>>
>>> Test config:
>>> sysctl -w net.smc.wmem=524288
>>> sysctl -w net.smc.rmem=524288
>>>
>>> Test results:
>>> SMC-R
>>> msgsize noautosplit autosplit
>>> 128KB 55.546 us 55.763 us
>>> 256KB 83.537 us 69.743 us (17% improve)
>>> 512KB 138.306 us 100.313 us (28% improve)
>>> 1MB 273.702 us 197.222 us (28% improve)
>>>
>>> SMC-D with smc-loopback
>>> msgsize noautosplit autosplit
>>> 128KB 14.672 us 14.690 us
>>> 256KB 28.277 us 23.958 us (15% improve)
>>> 512KB 63.047 us 45.339 us (28% improve)
>>> 1MB 129.306 us 87.278 us (32% improve)
>>>
>>> Signed-off-by: Guangguan Wang <guangguan.wang@xxxxxxxxxxxxxxxxx>
>>> ---
>>> Documentation/networking/smc-sysctl.rst | 11 +++++++++++
>>> include/net/netns/smc.h | 1 +
>>> net/smc/smc_sysctl.c | 12 ++++++++++++
>>> net/smc/smc_tx.c | 19 ++++++++++++++++++-
>>> 4 files changed, 42 insertions(+), 1 deletion(-)
>>>
>>
>> Hi Guangguan,
>>
>> If I remember correctly, the intention to use one RDMA-write for a possible large data is to reduce possible many partial stores. Since many year has gone, I'm not that sure if it would still be an issue. I need some time to check on it.
>>
>
> Did you mean too many partial stores will result in some issue? What's the issue?
>
>
>> BTW, I don't really like the idea to use sysctl to set the autosplit_size in any value at will. That makes no sense to improve the performance.
>
> Although 128KB autosplit_size have a good performance in most scenario, I still found some better autosplit_size for some specific network configurations.
> For example, 128KB autosplit_size have a good performance whether the MTU is 1500 or 8500, but for 8500 MTU, 64KB autosplit_size performs better.
>
> Maybe the sysctl is not the best way, but I think it should have a way to set the value of autosplit_size for possible performance tuning.
>
> Thanks,
> Guangguan Wang
>
Hi Wenjia,
Is there any update comment or information about this patch?
>>
>> Thanks,
>> Wenjia