--------------37A39D35466
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
I have put together the attached note that describes how
the device locking and queueing works for network drivers.
I'm interested to know whether I'm right or not, and whether
there are other significant wrinkles.
philip
-- Philip Gladstone +1 617 487 7700 Raptor Systems, Waltham, MA http://www.raptor.com/--------------37A39D35466 Content-Type: text/html; charset=us-ascii; name="skbuff.html" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="skbuff.html" Content-Base: "file:///F|/notes/skbuff.html"
<BASE HREF="file:///F|/notes/skbuff.html">
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
Ownership of sk_buffs and Network Drivers
There are a number of fields in the sk_buff which interact, and which the driver author needs to understand.
lock
The lock bit (actually a counter) is set when the IP layer passes the sk_buff to the driver. While it is set, the higher layers should not touch the buffer in any way. The default handling in a driver is to leave this flag alone, and hence a driver always runs with locked buffers. The downside of this approach is that the higher layers cannot remove buffers from any internal driver queues if they get acknowledged (or no longer need to be sent).
free
The free flag is set to indicate the disposition of the buffer when dev_kfree_skb is called. In a driver, this should not be modified as the higher layers are using it to control whether the dev_kfree_skb will actually free the buffer or not. TCP retransmits are handled by holding onto each buffer, and then retransmitting them -- obviously it is important that the driver does not free up these particular buffers! Once the ack is received, the buffer is either freed or marked as being freeable by dev_kfree_skb -- the choice depends on the lock and queue states.
On the transmit side, the free flag should not be modified by the driver. On the receive side, the free flag should be set to 1 before being passed up.
queue
The major forward and backward links can be used inside the driver to form lists of buffers. If the buffer is locked, then the higher layers will not touch this list. However, if the buffer is unlocked, then the buffer may be removed from a list asynchronously. This will happen if (say) TCP receives an ACK for the data in the queued buffer.
States
| lock | queue | Meaning |
|---|---|---|
| Yes | No | Buffer in use by driver. Higher layers will not modify it. Probably being transmitted. |
| Yes | Yes | Buffer on a queue inside driver. Probably waiting to be transmitted. Higher layers must not modify. Higher layers will not remove it from queue. |
| No | Yes | Buffer on queue either going into driver (managed by core/dev.c) or on a queue inside driver. Higher layers may remove buffer without notice from queue. Higher layers should not modify the buffer as it may be locked by the driver at any stage (even from interrupt level). |
| No | No | Dangerous state. If the driver knows about this buffer then it is (potentially) in trouble. The higher layers had better know about this buffer otherwise it will probably get lost. This is the only state when the buffer can be modified. |
Note that all these states can change at interrupt level. The only protection is that the unlocked, unlinked state can only be exited by higher layer action.
The driver can manipulate the buffers in any way, but should not unlock them.
The driver can unlock buffers, but must ensure that they are on a queue whenever they are unlocked. An unlocked buffer that is on a queue may be unlinked and freed at any time by the higher layer code.
The tcp retransmit code must be careful not to modify any buffer that is currently in use by a driver. A buffer can be in use if it is locked, or if it is not locked and on a queue. In the latter case, the retransmit code can unlink the buffer, modify it, and resubmit it.
To make the kernel behave as above, some small changes would need to be made.
The tcp_ack code needs to be modified as follows:
check to see if the buffer is locked. If so, then set the free flag to 3 (atomically). If not, then unlink it, and call kfree_skb. (Currently it always unlinks the buffer, even when it is locked.)
The tcp_do_retransmit code needs to be modified as follows:
After checking the lock flag on the buffer (and finding it unlocked), it should be unlinked. (Currently it waits until it is about to resubmit the packet before unlinking it. This violates the modification prohibition on queued buffers.)
--------------37A39D35466--