Re: [PATCH] net: add per device sg_max_frags for skb
From: Hannes Frederic Sowa
Date: Fri Jan 08 2016 - 06:48:18 EST
On 08.01.2016 10:55, Hans Westgaard Ry wrote:
On 01/06/2016 02:59 PM, David Laight wrote:
From: Hans Westgaard Ry
Sent: 06 January 2016 13:16
Devices may have limits on the number of fragments in an skb they
support. Current codebase uses a constant as maximum for number of
fragments (MAX_SKB_FRAGS) one skb can hold and use.
When enabling scatter/gather and running traffic with many small
messages the codebase uses the maximum number of fragments and thereby
violates the max for certain devices.
An example of such a violation is when running IPoIB on a HCA
supporting 16 SGE on an architecture with 4K pagesize. The
MAX_SKB_FRAGS will be 17 (64K/4K+1) and because IPoIB adds yet another
segment we end up with send_requests with 18 SGE resulting in
kernel-panic.
The patch allows the device to limit the maximum number fragments used
in one skb.
This doesn't seem to me to be the correct way to fix this.
Anything that adds an extra fragment (in this case IPoIB) should allow
for the skb already having the maximum number of fragments.
Fully linearising the skb is overkill, but I think the first fragment
can be added to the linear part of the skb.
David
When IpoIB handles a skb-request it converts fragments to SGEs to
be handled by a HCA.
The problem arises when the HCA have a limited number of SGEs less than
MAX_SKB_FRAGS.
(it gets a little worse since IPoIB need to yet another segment)
I have not found any easy way of fixing this with currenct codebase.
I think because of the complex forwarding nature, a global counter which
driver's can reduce during initialization time is the only solution I
see right now without changing the layout of the skb later on.
Unfortunately this doesn't resolve the cases were virtual machines
inject gso skbs, for those there still needs to be a slow path to do the
reformatting of the skb. :/
Bye,
Hannes