SNIFFING: NFR Performance Testing: Any Lacunae?

Hui Tsang (
Mon, 19 Apr 1999 22:24:20 +0530


I append another document that comments not so positively
on Linux's packet capture abilities.

I append how a company tested NFR on OpenBSD, BSD/OS, Solaris
and Linux, and what it reported. I append a heavily
edited document (so that it is easier for you to read the
relevant portions (BSD, BPF, Linux, and LSF).

The actual document is at:

Going by this document, we seem to get more for our investment
with OpenBSD on Intel, rather than Linux on Intel.

What do you have to say about this company's testing?
Do their February 1999 comments still hold?
Have they missed something, that could have improved
the performance of Linux?

Please let me know. Your inputs would help my organisation
choose an appropriate O/S for our packet capture application.


Caveat: Might have missed marking all the deleted portions with "<snip>"

                          NFR Performance Testing
                           Dug Song and Matt Undy
                           Anzen Computing, Inc.
                            Ann Arbor, MI 48106
                               February, 1999

In this paper, we attempt to loosely characterize NFR performance on various operating systems, comparing freely available and proprietary sniffing technology. Our results should be helpful in determining appropriate hardware/software configurations for successful NFR deployment. <snip> 2. Methodology

2.1. Hardware, software, and OS configuration The testbed setup consisted of a fast Ethernet (100baseT) closed network with the following five participants: * packet sender: 400 Mhz Pentium with 128 MB RAM running OpenBSD 2.4 * three hardware-identical NFRs: 180 Mhz Pentium Pro with 80 MB RAM running OpenBSD 2.4, BSD/OS 3.1, and Redhat Linux 5.1 (with 2.2.x kernel) * a comparable (?) Sun hardware NFR: 143 Mhz Ultrasparc with 128 MB RAM running Solaris 2.7 <snip>

2.3. Fast vs. Slow sniffing The commercial version of NFR includes proprietary fast sniffing modifications for BSD 4.4-derived operating systems (e.g. BSD/OS, OpenBSD) which read packets without performing additional memory copies. Slow sniffing on these systems uses the standard Berkeley Packet Filter, with increased buffer sizes. [MJ93] To implement fast sniffing on Solaris, we used an experimental membuf kernel module provided by Casper Dik [DIK99], which at the time provided the fastest sniffing available on that platform. The stock Solaris bufmod module provided the slow sniffing technology. On Linux, we used the Linux Socket Filter support in the Linux 2.2 kernel [TA99] to implement fast sniffing, as opposed to slow sniffing (without the LSF). <snip> 3. Results

With the slower traffic, we found differences between slow and fast sniffing on each of the platforms to be roughly even - somewhere on the order of 20% improvement with fast sniffing enabled. But even with this modest traffic load, all platforms in slow mode experienced drops. Only the BSDs were able to pick up 100% of the traffic in fast mode. With increased traffic (47 Mbps or fast Ethernet running at roughly 50% load), the gap widens. Fast mode on the BSDs comes out far ahead of the pack, with Linux sniffing less than 10% of the traffic on the wire, and Solaris, at its best, doing roughly 70%. Still, Solaris' membuf fast mode shows tremendous improvement over the stock Solaris bufmod, about 3 times better. <snip>

3.2. NFR For the NFR tests, we used the slower traffic to allow for a more representative picture of performance differences between the OSs (since Linux lags so far behind with the faster traffic). Without any filters running, NFR in fast mode on the BSDs still manages to capture all the traffic on the wire, with fast mode Solaris a close second. IP fragment and TCP session reassembly don't seem to cost too much - at least with the data sets we used. When the default NFR filter set is enabled, things get even flatter - with the surprising exception of fast mode OpenBSD. At this point, we expected performance on the BSDs and Linux to roughly match, due to I/O considerations. But OpenBSD in fast mode appears to be the clear winner, for reasons we haven't been able to determine yet (yes, our results were repeatable!). <snip> 4. Conclusions and Future Work

4.1. OS choice Much to the chagrin of Linux users on the NFR mailing lists, we found that Linux performs poorly - worst, in fact, among the operating systems tested - due to its inefficient packet handling (with or without LSF). [AL99] Only Windows NT fared worse in early beta testing; NFR has subsequently chosen not to support it as a platform for commercial release. The experimental membuf module for Solaris shows some promise, however, and might even make Solaris a contender against the BSDs, if run on appropriate hardware. But for almost all environments, it's OpenBSD we recommend, based on performance considerations alone. <snip> 5. Acknowledgements

<snip> Casper Dik (Sun security engineer) provided us with an early alpha of his membuf module for fast sniffing on Solaris. Tyler Allison provided us with the libpcap modifications necessary for NFR to work on Linux 2.2 with LSF. 6. References

[TA99] T. Allison, pcap-linux.c,, Feb. 1999. <snip> [AL99] A. Lambeth, "Linux packet capture", mailing list, l, Feb. 1999. <snip>

To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to