Re: tlug: Clustering Linux Boxes

["Manuel M. T. Chakravarty" <chak@example.com>]

"Austin Kurahone" <a_downunder@example.com> wrote,

> I was just thinking about what to do with my time now that I finished the 
> 0.2a relase of GLF[1], and I was thinking about implementing a parallel 
> processing cluster with all the computers that I have.
> 
> Now the question that I have is that will making a cluster(eg: Avalon with 
> Extreme Linux) give me performace advantages for all my programs, or do they 
> need to be specially coded to take advantage of the
> cluster.  

The parallel programming HOWTO

  http://metalab.unc.edu/LDP/HOWTO/Parallel-Processing-HOWTO.html

will help you get started.  But be warned, parallel
programming is still very much a research thing.  There are
a million approachs, most are tedious and restricted to
programs/algorithms that exhibit a particular parallel
behaviour.  The upside of this is that a good idea can still
make you famous in this area :-)

The most simple approach to programming a cluster is message
passing with MPI or PVM (as already mentioned by Chris and
Steve).  IMHO, don't waste your time with PVM, but use MPI.
I recommend the LAM implementation:

  http://www.mpi.nd.edu/lam/

The good thing about MPI is that it is relatively simple;
the bad thing is that its level of abstraction for
specifying parallel computations is comparable to what
assembly language is for sequential computations.

> If they need 
> to be specially programed, what is the feasibility of coding/implementing a 
> transarent layer(a module maybe) that will allow the Linux kernel to 
> automatically distribute the separate threads/tasks to the various nodes in 
> the cluster???  Or do I have no idea what I'm talking about ;-)

Check out 

  http://www.mosix.cs.huji.ac.il

They are doing exactly this.  The disadvantage of such an
approach is that it only works for applications that are
rather coarse grained, i.e., the threads are computation
intensive and require relative little communication.  For
other kinds of applications, your machines will have so much 
load balancing and communication work to do that they do
little else -- its like having a machine trashing its swap 
area.  (The good thing is, of course, that for those
applications for which it works, it is quite painless.)

Steve wrote,

> Also, there are (eg) FORTRAN compilers which will automatically take
> advantage of a certain amount of parallelism that is inherent in
> things like do loops.  

This only works for vector computers and clusters with very
low-latency (= expensive) networks and always requires the
code to be written such that the compiler can find
sufficient parallelism (in your average sequential program,
there is not enough of it).

> I doubt there are C compilers that do this (at
> all, let alone free code), but you could probably hack an optimization
> stage into GNU C/C++.

By now, many supercomputer vendors have a language
independent vectorizer or parallelizer, which you can
include in Fortran as well as C compiles via the right
command line switches.  But sometimes you have to use
special pragmas or language extensions to make full use of
these compiler stages.

There is even a HPC effort that tries to do to C what HPF (= 
High Performance Fortran) did to Fortran 90/95, but then it
seems as if HPF is dying anyway.

<shameless plug>
  Now, if you have too much time, you could think about
  having a look at one of the research projects I am
  involved in.  We are writing a compiler for, so-called,
  nested data parallelism, which allows to code a wide range
  of parallel programs in a portable way (more portable and
  much easier to program than MPI):

    http://www.score.is.tsukuba.ac.jp/~chak/nepal/

  The problem is that the compiler is still in the
  development phase, but it also targets clusters (in fact,
  we will soon put our runtime and communication system up,
  which also works ontop of MPI-2).  The code is open and
  GPL, so everybody who wants to tinker is welcome to do so.
</shameless plug>

Cheers,

Manuel
-------------------------------------------------------------------
Next Nomikai: May 26 (Wed), 19:00 Tengu TokyoEkiMae 03-3275-3691
*** Special guest: Eric S. Raymond
Next Technical Meeting: June 19 (Sat), 12:30  place: Temple Univ.
Topic: Linux SMP and the Intel SC450NX Quad Xeon Server platform
-------------------------------------------------------------------
more info: http://www.tlug.gr.jp        Sponsor: Global Online Japan