Re: SMP scalability: 8 -> 32 CPUs

Zow (
Sat, 05 Dec 1998 22:48:12 -0800

> > BTW, ASCI Blue Pacific, the 1344-processor IBM SP/2 version, is a cluster
> > of 8-way SMPs (iirc). It has a pretty horrible memory latency problem
> > inside a single node (it's as fast as talking to another node), but the
> > _idea_ is to have a lot of really fast local memory, plus a trusted, fast
> > connection to other SMPs.
I don't know about latency, but I can tell you that the communication
bandwidth bites. Our code is currently only written for MPI and if we run four
processes on a node, performance is great (MPI is smart enough to used shared
memory). If we use eight processors, the bandwidth between the two nodes kills
us. I'm sure it's because there may be as many as 16 channels of communication
there (nodes^(procs/node)). I'm now looking at using threading as using that
and going down to one process per node should help quite a bit. Now this is
just what we've seen in the group I work for; YMMV. What does this have to do
with Linux? I prototype on my Linux workstation. As our project gets more
involved, I will probably be running on Linux clusters and I want to be able
to have a feel for how our code will run on Blue.

> 4-way machines according to the information I found. Looks like they have
> a new "machine" on their way, with a total of 5856 CPU's (1464 4-way SMP
> machines) for "closed side" (classified stockpile stewardship computing ?)
Can't say. It's classified. And I don't have my security clearance yet. What
you have to keep in mind is that the machines on the closed side are
completely secure. There are no physical connections between the machines on
the closed side and any other networks (as those other networks may,
potentially, be connected to the 'Net). To transfer files between the open and
closed side, an operator must physically move a magnetic tape. So while the
specs on Blue are impressive, it's really two seperate machines: they can't be
run in tandem.

> From the information it looks like they have 2/3 of these already, which
> should put that relatively high on top500 list if it had been included.

Due to the security regulations, that would have been improbable.

> Both uses 4-way 332 MHz PPC604e, but there were old notes about planning
> to move to Power-3 in "1998". Looks like that have been delayed thought.
Some of the nodes are still the old Power-2 nodes (UP), but those should all
be gone shortly (maybe they have been and I missed the announcement). From
what I understand, the Power-3 upgrade is "real soon now".

"Zow" Terry Brugger

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to
Please read the FAQ at