User-space Networking (Was; unusual startup messages)

Craig Milo Rogers (rogers@isi.edu)
Fri, 01 Nov 96 14:31:01 PST


>> >Yes, you can do it in user space, but your performance will suck unless you
...
>> Actually, the hardware needn't be that special. It mainly
>> needs to identify "normal" incoming TCP and UDP packets, and store
...
>Yes, you can get _some_ networking going that way, but you sure as hell
>aren't going to get UNIX semantics. Another favourite pastime of
...

You have raised some very cogent points. To save space, I
have not quoted your message in detail. Instead, I'll focus on some
overall application and OS design issues involved (and that way I'll
also feel a little better about using bandwidth on linux-kernel).

The first question to ask is, "Why would I want a user-level
TCP/IP implementation?" Some reasonable motivations are: "It will
increase system robustness", "It will increase system security", and
"It will increase system performance". Of course, whenever you select
one of these motivations, you have to demonstrate that you actually
meet the stated goal.

I'll select "It will increase system performance", in a
specific context: a dedicated Web/FTP/NSF/DNS server. In the
hypothetical dedicated Web server process, for instance, I'll run
special code whose purpose is to grab HTML data from storage and shove
it out the net as efficiently as possible. In this instance,
maintaining campatibility with the existing Unix networking API is not
terribly important, and may be sacrificed in the interests of overall
performance.

"But wait", you may reply. "A Unix system, even a dedicated
Web server, must run dozens of complicated network-dependent
processes, and you probably wouldn't have resources to rewrite them
all!" This is an entirely valid objection, and I must address it.

One approach is to implement user-level TCP/IP in the
dedicated Web/FTP/NSF/DNS server applications while retaining the
kernel-level TCP/IP for use by more general Unix networking processes.
Admittedly, now you have *two* (or more) independent TCP/IP stacks to
maintain, and it doesn't satisfy the goal of the initiator of this
discussion, which to question whether the TCP/IP stack needs to be in
the kernel at all. Nonetheless, this approach will probably meet my
goal of a creating a very high performance Web/FTP/NSF/DNS server.

Another approach is to implement user-level TCP/IP in the
dedicated server processes, and have a user-level TCP/IP process act
as a daemon (ala kerneld) for all other network-using processes. It
is fairly easy to imagine how to implement this and preserve full Unix
semantics for the processes that require them. The cost is fairly
high, of course, due to the multiple user/kernel/user context switches
required to pass data and control between processes. However, *if*
the bulk of system processing is in the dedicated server processes,
which are not subject to these context switches, then the increased
overhead in the other processes may be negligible to the system as a
whole.

Another approach for supporting the "general Unix networking"
processes is to use a modified libc that intercepts the network calls
and uses, say, shared memory, to communicate with a seperate (thus
crash-resistant) TCP/IP daemon process. It may be possible this way
to avoid some unwanted data copies, although there may still be an
inter-process IPC call needed to replace many of the the kernel calls
of a "normal" Unix implementation; as the literature of microkernels
demonstrates, it is difficult to do this efficiently.

I have used the goal of a "high-performance Web/FTP/NSF/DNS
server" in my discussion above. I believe I could provide similar
arguments in favor of a user-space TCP/IP implementation for the goals
of increasing system robustness or security. Of course, what I have
provided as "proof" is merely hand-waving; the success of the Internet
has been based upon performing concrete experiments rather than
gedanken ones. Nonetheless, I hope that I have supported the point
that the proper analysis is to ask whether the (purported) benefits of
a user-level TCP/IP implementation do indeed outweigh the requisite
costs in a system context, rather than to dismiss the concept as
infeasible based upon cost alone.

Craig Milo Rogers