Re: Dropping NETIF_F_SG since no checksum feature.
From: Michael S. Tsirkin
Date: Thu Oct 12 2006 - 15:13:07 EST
Quoting r. David Miller <davem@xxxxxxxxxxxxx>:
> Subject: Re: Dropping NETIF_F_SG since no checksum feature.
>
> From: "Michael S. Tsirkin" <mst@xxxxxxxxxxxxxx>
> Date: Wed, 11 Oct 2006 23:23:39 +0200
>
> > With my patch, there is a huge performance gain by increasing MTU to 64K.
> > And it seems the only way to do this is by S/G.
>
> Numbers?
>
I created two subnets on top of the same pair infiniband HCAs:
root@sw069 ~]# ifconfig ib0
ib0 Link encap:UNSPEC HWaddr
00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:12.4.3.69 Bcast:12.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::202:c902:20:ee45/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
RX packets:1382531 errors:0 dropped:0 overruns:0 frame:0
TX packets:2725206 errors:0 dropped:5 overruns:0 carrier:0
collisions:0 txqueuelen:128
RX bytes:71892772 (68.5 MiB) TX bytes:5290011992 (4.9 GiB)
[root@sw069 ~]# ifconfig ibc0
ibc0 Link encap:UNSPEC HWaddr
00-03-04-06-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:11.4.3.69 Bcast:11.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::202:c902:20:ee45/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65484 Metric:1
RX packets:115647 errors:0 dropped:0 overruns:0 frame:0
TX packets:253403 errors:0 dropped:4 overruns:0 carrier:0
collisions:0 txqueuelen:128
RX bytes:6014720 (5.7 MiB) TX bytes:16589589008 (15.4 GiB)
The other side was configured with 12.4.3.68 for MTU 65484
and 11.4.3.68 for MTU 2044.
And then I just run netperf:
[root@sw069 ~]#
[root@sw069 ~]# /mswg/work/mst/netperf-2.4.2/src/netperf -f M -H 12.4.3.68 -c -C
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 12.4.3.68 (12.4.3.68)
port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % S us/KB us/KB
87380 16384 16384 10.00 286.45 40.20 25.28 5.482 3.448
[root@sw069 ~]# /mswg/work/mst/netperf-2.4.2/src/netperf -f M -H 11.4.3.68
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.4.3.68 (11.4.3.68)
port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. MBytes/sec
87380 16384 16384 10.01 782.55
This is all very preliminary - but I hope you get the idea -
increasing MTU is very helpful for infiniband, and infiniband adapters
handle large S/G lists without problems, but the verbs
do not include support for IP checksums, so these must be done in software.
So what we would like, is for the infiniband network device to say
"I don't support checksums, I only support S/G" and then for
network layer to do the checksumming for us piggybacking on data copy
at least for cases where it does perform the copy.
Does this makes sense now?
--
MST
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/