Re: Lets get this right (WAS RE:MOSIX and kernel mods)

Richard Gooch (rgooch@atnf.csiro.au)
Sun, 7 Mar 1999 16:55:10 +1100


Michael Loftis writes:
> On Sat, 06 Mar 1999 17:57:51 -0500 Jeff Millar <jeff@wa1hco.mv.com>
> wrote:
>
>
>
> > Within the next year, we'll have networked computer chips that
> > use 1-4 Gbit per second serial links between them. Imagine
> > a SIMM/DIMM kind of thing holding 8 CPU chips each with 64 MB on die
> > ...plug in as many as you like on your motherboard. The interconnect
> > protocol uses DSM so it looks like an SMP.
>
> Which is exactly why itd be good to start work now. Nearby is a lab
> w/ 100Mbit ethernet and Giga-Ethernet. Both are very fast, and
> aside from using a Paragon backplane, the Giga is about as fast as
> it gets. This is *now* in 5 years Giga Ether will be consumer
> product. Imagine a 16-node Dual P-II Linux cluster on a switched
> full-duplex gigabit network... Right now pretty spendy (if you
> forgive the fact that there isn't free clustering in the kernel) but
> it *is* possible. Flash forward five years when the P-II can be
> bought at a garage sale and a Gigabit Ether switch would cost $50.

OK, let's look at some numbers and compare. 100 Mb/s EtherNet is now a
commidity item: Gb/s is still experimental. So we have networks with
10 MB/s bandwidth and 2 millisecond latencies.

Now let's look at the bandwidth of a modern, commidity computer, say
PII. With 100 MHz SDRAM, a 4-1-1-1 burst gives you 8 bytes per step;
with 4 steps that's 32 bytes for the burst. It takes 7 cycles. Let's
say 8 to make the maths simpler. So 8 cycles gives 32 bytes. That's 4
bytes per cycle: 400 MBytes/s. The latency is going to be well under
200 nanoseconds.

So: we have a factor of 40 in bandwidth and 10000 in latency. And this
comparison is only done with the boring 64 bit bus in your average PC.
It gets worse if you look at a 256 bit bus: bandwidth goes to 1.6 GB/s,
which is 160 times better than current EtherNet.

Right now, there's no competition: networks are so much slower than
what you find in a computer.

Now, I hear some of you saying that networks are getting faster all
the time. But so are computers! Let's look back a bit:

6 years ago, a 486 DX266 was hot stuff, but still a commidity
item. IIRC, bandwidth back then was 20 MB/s. Hm. That's a factor of 20
from where we are now. Moore's Law seems to be holding up (he predicts
16).

6 years ago, I had the pleasure of hacking on a VX/MVX system. This
was a distributed memory machine costing $50000 and up. The
interconnect between the boards was rated at 320 MB/s. What was
"leading edge" back then is now available for the price of an average
pushbike.

And 6 years ago, where was networking? 1MB/s (10Mb/s), that's
where. In fact, it was there in 1989 too: 10 years ago. It's taken
around 10 years to get a factor of 10 improvement in bandwidth. And I
don't recall the latency changing much over that time, either. 6
years ago, all we had were whispers of FDDI. $50000 wasn't going to
buy you something 16 times faster (like how the VX/MVX was faster).

Networking has not kept up with Moore's Law. It seems computers have,
however. So the logical progression is that in the future, we're going
to see more of the same. It's going to take years before gigabit
EtherNet is affordable, and by then computers will be even further
ahead.

> Heck even if this view is off and skewed to heck, *it is possible*.
> And things are getting faster, it's only a matter of time before
> networking is fast enough (or integrated enough) that clustering
> such as this would be possible.

Let me make a bottom-line statement: networking will *never* be as
fast as internal computer speeds. Technology trends point to computers
increasing at a greater rate. And then there's physics. Unless you're
planning on breaking the speed of light, it's always going to take a
lot longer to send information across a room than across a chip.

[...]
> Think.

Count.

Regards,

Richard....

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/