Mariah Cluster Details

(Shaheen Tonse.)


Here are some pictures and notes on the current (6/29/2004) configuration of mariah.
We currently have 2 racks, one full and one empty. We bought the 2nd one in anticpation of buying more nodes. The full rack has 13 nodes (1 Master and 12 slaves.) Each slave has 2 processor elements (PE).
Below is a front view. Starting from the bottom are the 12 slaves. The bottom 2 were bought later and look a little different. After 6 slaves is a gap where the fold-in monitor screen is located. Above that are 6 more slaves and then the Master node. Above that is the Ethernet switch, I believe. To the left of the rack is the empty rack;  you can see it through the door.


Front of full rack

The next 2 pics are just close-ups of the front again
Close up of front top   Front bottom

The back view of the same rack:

Back view full rackOnce again you can see the 12 slaves starting from the bottom, then the master, the ethernet and at the top with the orange cables, the Myricom. The Myricom will accept another 19 nodes, currently 13 out of 32 are used. More than that and we will have to buy extra Myrinet blades. Also, according to Gary Jung the Ethernet switch (pinkish cables, blue crate, below Myrinet, and above Master) is using 13 of the 24 available ports.
Back view, top half


The bottom half of the back view:  Bottom of back

Any new nodes we buy will go on the empty rack:       Empty rack, front
Gary put 3 spare crates into the empty rack, so that somebody would not assume it was "spare" and steal it! It looks to me like there is room for about 36-40 nodes,if all we were putting in were nodes. So we will probably end up being constrained by $$$ rather than lack of space. (The new nodes are thinner than the existing ones on the full rack by  a factor of 2.) The pics below are of another cluster that has the new single-thickness nodes, a single node closed, and a single node opened up.

Other cluster with thin nodes          Closed node     Open node  



Here are more pictures of the empty rack, the front, top portion:
empty rack front, top


and the back view:
Empty rack, back


And finally, a personal touch to the cold, impassionate world of Linux cluster computing:

aaaaa


aaaaa



Some $$$ details:
The price per node will be about: $1750 (node) + $600 (Myrinet) + $300/year (upkeep) = $2700 for 1 year
or $3000 for 2 years. (I do not know how much of the upkeep we can or should pay in advance.)

If we buy more than 19 nodes we will need a new Myrinet switch, $2000.
If we buy more than 11 new nodes we will need a new ethernet switch (~$800).
If we put single-thickness nodes, Gary recomends getting a terminal server ($3000) to reduce the amount of cabling that normally goes through KVM. This is quite a bit of money, so I suggest we consider whether or not to get it depending on the number of nodes we buy and think we will buy in the future.