RE: [Xen-devel] "tcp: refine TSO autosizing" causes performance regression on Xen

[David Laight]

From: George Dunlap
> Sent: 16 April 2015 09:56
> On 04/15/2015 07:19 PM, Eric Dumazet wrote:
> > On Wed, 2015-04-15 at 19:04 +0100, George Dunlap wrote:
> >
> >> Maybe you should stop wasting all of our time and just tell us what
> >> you're thinking.
> >
> > I think you make me wasting my time.
> >
> > I already gave all the hints in prior discussions.
> 
> Right, and I suggested these two options:
> 
> "Obviously one solution would be to allow the drivers themselves to set
> the tcp_limit_output_bytes, but that seems like a maintenance
> nightmare.
> 
> "Another simple solution would be to allow drivers to indicate whether
> they have a high transmit latency, and have the kernel use a higher
> value by default when that's the case." [1]
> 
> Neither of which you commented on.  Instead you pointed me to a comment
> that only partially described what the limitations were. (I.e., it
> described the "two packets or 1ms", but not how they related, nor how
> they related to the "max of 2 64k packets outstanding" of the default
> tcp_limit_output_bytes setting.)

ISTM that you are changing the wrong knob.
You need to change something that affects the global amount of pending tx data,
not the amount that can be buffered by a single connection.

If you change tcp_limit_output_bytes and then have 1000 connections trying
to send data you'll suffer 'bufferbloat'.

If you call skb_orphan() in the tx setup path then the total number of
buffers is limited, but a single connection can (and will) will the tx
ring leading to incorrect RTT calculations and additional latency for
other connections.
This will give high single connection throughput but isn't ideal.

One possibility might be to call skb_orphan() when enough time has
elapsed since the packet was queued for transmit that it is very likely
to have actually been transmitted - even though 'transmit done' has
not yet been signalled.
Not at all sure how this would fit in though...

	David