Re: SNIFFING: How do you evaluate this article?

Andi Kleen (ak@muc.de)
20 Apr 1999 07:57:10 +0200


In article <19990419220105.A3236@nm.ece.iisc.ernet.in>,
Hui Tsang <huitsang@ece.iisc.ernet.in> writes:
> All:
> I append a message from the NFR (Network Flight Recorder)
> Users mailing list.

> The company (NFR) decided to not support Linux for its
> sniffing product. One of their software engineers
> explains.

> Does the evaluation still hold?
> Are there any mistakes?

His evaluation is basically correct, but his conclusion is wrong.

I also object to (2):

A read() is cheap on Linux, and it is possible to overcome
the latency problem by using multiple threads to read from the socket.

My guess (I have no locked at NFR architecture too closely) is that
they do something like this:

process:
read(packet_socket)
do some work on the packet
hand it over to someone else

This does not get the data from the kernel at high rates. A better architecture
for really high speed tapping on linux most likely would do:

multiple threads:
read(packet socket)
put it into a shared queue

thread:
get packet from queue
do some work
hand it over to someone else.

This does the same amount of copies: BSD does the copy in kernel space in
BPF, Linux at the kernel->user read, just BSD did tolerate the latencies
better. I believe that multithreaded reader design would make Linux
look better. I also bet they didn't bother to increase the packet socket's
buffer size. With all that packet copying going on the small context
switch overhead between the threads and the network bottom half is negligible.
read() itself on Linux is really cheap, Linux generally has much less VFS
overhead than BSD or SYSV.

Alexey's turbo packet capture patches are fast, but the implementation
is disgusting: a very ugly hack that would most likely never end up in an
official kernel (except when we kill Linus first ;). If you are willing
to eat the memcpy overhead (which BPF on BSD does AFAIK), then I don't
believe the additional small speed gains possible over a proper
multithreaded reader with more kernel hacks are worth the effort.
It is all left in the noise compared to the copying anyways.

Linux packet capture is different. If they just port their BSD code over
without changes they shouldn't expect to see good performance.

Regarding missing counters for dropped packets: I did most of the work
for that in late 2.1, but it was too late to add for 2.2. It really is
only a small, not impacting patch, it is not difficult. Expect it in early 2.3

I hope this helps,

-Andi


> I append the message. I would appreciate a point-wise
> comparison (if possible).

> Please let me know.
> Thanks.

> Dinesh


> http://www.nfr.net/nfr/mail-archive/nfr-users/1999/Feb/0110.html
> --
> Linux packet capture (was RE: NAI Security Advisory: Vulnerability in NFR
> 2.0.2-Research)

> Andrew Lambeth (andrew@nfr.net)
> Sat, 20 Feb 1999 00:08:06 -0500

> * Messages sorted by: [ date ] [ thread ] [ subject ] [ author
> ]
> * Next message: Jan B. Koum: "Re: NAI Security Advisory:
> Vulnerability in NFR 2.0.2-Research"
> * Previous message: dustin sallings: "Re: NAI Security
> Advisory: Vulnerability in NFR 2.0.2-Research"
> * In reply to: Derek Mezack: "Re: NAI Security Advisory:
> Vulnerability in NFR 2.0.2-Research"

> ------------------------------------------------------------------

> I downloaded a copy of the latest Linux kernel sources I could find
> (2.2.1) and had a look to see how the packet capture inadequacies have
> been addressed. Here are my findings after a cursory examination of
> the code:

> Observations:

> 1. The 2.2 kernel includes a bpf-like filtering capability, called LSF
> (Linux Socket Filter.) This capability allows bpf-style filters to be
> applied to packets off the wire in the kernel. Since this means that
> only packets which pass the filter will be copied up to user-space,
> applications which need to see some *subset* of all packets on a
> network should see a dramatic increase in performance.

> 2. As far as I can see, no facility has been added for getting multiple
> whole packets (including the LLC header) out of the socket. (i.e. you
> still have to make a call to read() for each individual packet.)

> 3. Because the code in linux/net/packet/af_packet.c doesn't see each
> packet until after it has been massaged by the device driver to obscure
> the LLC header, it must "unmassage" it to include LLC info in the
> packet.

> 4. Though I admit that I didn't look very hard at this one, it still
> seems to me that there is no way to determine how many packets were
> dropped because the calling application was unable to handle them fast
> enough. They would appear to be discarded silently in
> linux/net/packet/af_packet.c after calls to sock_queue_rcv_skb() fail.

> (As I said, these observations are the result of a relatively shallow
> examination of the Linux kernel code, so anyone who is more familiar
> with Linux, please correct me if you believe that I have overlooked or
> misunderstood something in making them)

> Conclusions:

> Unfortunately, #1 by itself doesn't do anything to help applications
> like NFR which are interested in seeing *all* packets on a network in
> most cases. (i.e. it doesn't matter how good the filter implementation
> is if you are not filtering ;) As an example: there will likely be a
> marked performance increase with an LSF Linux kernel vs. a non-LSF Linux
> kernel when you do something like `tcpdump host foo.bar.baz` but no
> performance increase when you just do `tcpdump`.

> Contrast #2 with BPF, in which case a single call to read() can yield
> up to 512K bytes of packet data (or more if you further increase the
> buffer sizes.) This means that compared to a *BSD kernel, a Linux
> kernel will have to context switch about 1000 times more frequently
> when running a sniffing application. Also note that this shortcoming
> affects even applications which use LSF to filter out uninteresting
> packets, since the packets they do get will still come up to user-space
> one at a time.

> The operations discussed in #3 are not trivial in terms of CPU usage
> since they must be performed for every single packet. BPF does not
> have this problem because it is implemented via hooks into the device
> drivers.

> #4 is bad for obvious reasons. A sniffer must be able to report how
> many packets *it* (not just the interface or abstractions below it)
> dropped.

> I don't know what policies the Linux development community has
> regarding what gets added/changed when, but I imagine that the changes
> required to fix #2 and especially #3 above will be non-trivial
> enough to not be included until 2.3 (and wouldn't be available in a
> stable rev until 2.4 or 3.0, I guess). Since 2.2 was released less
> than a month ago, I think that for the foreseeable future our
> recommendation will continue to be to avoid Linux as a sniffing
> platform for performance reasons.

> At this point I would like to ask that we end the Linux/*BSD debate on
> nfr-users.

> --
> Andrew Lambeth - Software Engineer, Network Flight Recorder, Inc.

> ------------------------------------------------------------------
> -
> To unsubscribe from this list: send the line "unsubscribe linux-net" in
> the body of a message to majordomo@vger.rutgers.edu

-
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@vger.rutgers.edu