Re: [net-next RFC] pktgen: don't wait for the device who doesn't free skb immediately after sent

[Jason Wang]

On Monday, December 03, 2012 08:01:11 AM Stephen Hemminger wrote:
> On Mon, 03 Dec 2012 14:45:46 +0800
> 
> Jason Wang <jasowang@xxxxxxxxxx> wrote:
> > On Tuesday, November 27, 2012 08:49:19 AM Stephen Hemminger wrote:
> > > On Tue, 27 Nov 2012 14:45:13 +0800
> > > 
> > > Jason Wang <jasowang@xxxxxxxxxx> wrote:
> > > > On 11/27/2012 01:37 AM, Stephen Hemminger wrote:
> > > > > On Mon, 26 Nov 2012 15:56:52 +0800
> > > > > 
> > > > > Jason Wang <jasowang@xxxxxxxxxx> wrote:
> > > > >> Some deivces do not free the old tx skbs immediately after it has
> > > > >> been
> > > > >> sent
> > > > >> (usually in tx interrupt). One such example is virtio-net which
> > > > >> optimizes for virt and only free the possible old tx skbs during
> > > > >> the
> > > > >> next packet sending. This would lead the pktgen to wait forever in
> > > > >> the
> > > > >> refcount of the skb if no other pakcet will be sent afterwards.
> > > > >> 
> > > > >> Solving this issue by introducing a new flag IFF_TX_SKB_FREE_DELAY
> > > > >> which could notify the pktgen that the device does not free skb
> > > > >> immediately after it has been sent and let it not to wait for the
> > > > >> refcount to be one.
> > > > >> 
> > > > >> Signed-off-by: Jason Wang <jasowang@xxxxxxxxxx>
> > > > > 
> > > > > Another alternative would be using skb_orphan() and skb->destructor.
> > > > > There are other cases where skb's are not freed right away.
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > > > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > 
> > > > Hi Stephen:
> > > > 
> > > > Do you mean registering a skb->destructor for pktgen then set and
> > > > check
> > > > bits in skb->tx_flag?
> > > 
> > > Yes. Register a destructor that does something like update a counter
> > > (number of packets pending), then just spin while number of packets
> > > pending is over threshold.
> > 
> > Have some experiments on this, looks like it does not work weel when
> > clone_skb is used. For driver that call skb_orphan() in ndo_start_xmit,
> > the destructor is only called when the first packet were sent, but what
> > we need to know is when the last were sent. Any thoughts on this or we
> > can just introduce another flag (anyway we have something like
> > IFF_TX_SKB_SHARING) ?
> 
> The SKB_SHARING flag looks like the best solution then.
> Surprisingly, transmit buffer completion is a major bottleneck for 10G
> devices, and I suspect more changes will come.

It works, but we may lose some chances to use clone_skb and stress the device 
and driver more. I'm thinking maybe we can turn back to my original RFC to 
introduce another flag. This flag maybe also useful for BQL and zerocopy in the 
future since both of them are sensitive to the transmit buffer completion. 
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/