Logging http farms/packet sniffing

From: G . Sumner Hayes (SumnerH@fool.com)
Date: Thu Mar 02 2000 - 17:35:47 EST


I'm interested in logging all of the http requests to a farm of http
servers. For various reasons I can't install software on the servers
themselves, so I'm looking at a promiscuous packet-sniffing approach.
The logger would be a Linux machine(or machines) sitting in front of the
farm where they'd have access to all the network traffic. This is to be
a quick and dirty hack that only needs to be in place for a few days,
but it needs to be written and running ASAP.

I can throw some pretty beefy servers at the problem, maybe up to a
half-dozen dual 600Mhz Intel machines with 512MB RAM if needed (though
I'd imagine the required resources would be considerably less, probably
well within the capabilities of a single machine). I'm looking at order
300 hits/second to the farm (peak) at the moment, though the ability to
scale beyond that would be nice as well.

I'm potentially interested in longer-term solutions (which I could spend
more than 10 days or so implementing) as well, but a short-term solution
is first priority.

Right now I'm contemplating one of:

tcpdump: Can tcpdump keep up with ~30Mbit/sec of traffic? Most of the
traffic is outgoing, and it's just the incoming http requests that I'm
interested in. How should I handle TCP stream reassembly if I go with
tcpdump? Doing the reassembly as a post-processing step is fine if
there's something out there that can handle it.

Super Sniffer (http://www.mobis.com/ajax/projects): This is another
sniffer that does all of the TCP stream reassembly for me. Anyone have
any experience with it and know if it can handle tens to hundreds of
hits per second? It seems a bit like hax0rware to me, so I'm not sure
it's really been stress tested, but doing the stream reassembly is a
big win.

libpcap: Obviously requires that all the reassembly be done by hand.

Anyone have any experience with this sort of thing? Suggestions about
other tools to use or gotchas to look out for would be gratefully

Thanks for your time,


rage, rage against the dying of the light
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@vger.rutgers.edu

This archive was generated by hypermail 2b29 : Tue Mar 07 2000 - 21:00:27 EST