Re: tlug: Karl-Max has cool dreams [was: dual-pentium processors]

["Manuel M. T. Chakravarty" <chak@example.com>]

"Stephen J. Turnbull" <turnbull@example.com> wrote,

> >>>>> "Manu" == Manuel M T Chakravarty <chak@example.com> writes:
> 
>     Manu> "Stephen J. Turnbull" <turnbull@example.com> wrote,
>     >> >>>>> "Karl-Max" == Karl-Max Wagner <karlmax@example.com>
>     >> writes:
>     >> 
>     Karl-Max> By 2020 your computer will house a 64 paralell processor
>     Karl-Max> chip
>     >> The other stuff is one-dimensional extrapolation as far as I
>     >> can see, and (possibly) conservative.  Why do you think this
>     >> will take so long, and why do you think it will be as few as
>     >> 64?
> 
>     Manu> The question is, will it help you much?
> 
> Dunno.  I'd bet on it, though.

I won't.  I mean, if the next 20 years of research in
parallel processing are as successful as the last 20, you
won't gain much with 64 processors.

> 
>     Manu> There are not many people who can write a program that
>     Manu> scales to 64 processors, and if you can, it is an aweful lot
>     Manu> of work.
> 
> Who needs to?  `M-! ps ax | wc' ==> `54   320   2233'
>
> I've got 54 processes running, 

Yeah, 95% of them sleeping.  Won't help you much utilizing
your CPUs.  A better measure is the CPU load of your machine
(i.e., average number of processes in the ready-to-run
queue).  I guess you are not really often above 1, 2, or
maybe 3 processors.

> and several of them (X server,
> Netscrap, XEmacs) could very usefully be multi-threaded.  Processor
> per thread!  Wasteful?  Sure---but I know lots of people who only use
> half their RAM and half their hard drive.  If multiple processors
> could be packed as cheaply as bytes are today (Think Big!), there
> would be no issue about scheduling, interrupts, etc---just use any
> free processor.

Wastefulness is not necessarily the problem and more
processors don't necessarily make things faster.  A typical
effect on a parallel program is that it gets faster with
more processors up to 16 nodes, and then, with 32 it is just
as fast as with 2 and with 64 it is slower than with a
single one.  Why?  Because the processors spend more time
sending data back and forth and waiting for each then doing
actual work.

In a parallel machine you have two choices: shared memory
(all processors access the same memory chips) or distributed
memory (each processor has its own piece of memory).  Either 
way, you get into serious trouble with increasing numbers of 
processors:

Shared memory: These days even a single processor is so much 
  faster than the memory and bus that you have to have
  caches and stuff to make the gap acceptable.  With more
  processors that gets worse.  If you hook to many onto one
  bus, none of them gets its memory requests satisfied
  reasonably fast.

Distributed memory: More processors mean more communication
  and communication is even slower than local memory
  access.  You have to organize your program very cleverly
  to get any speed up beyond 16 processors -- just
  multi-threading things doesn't help anything.

(You can mix both approaches, of course.  Then, you get a
mix of the problems.)

> (Yes, that's overly simplistic; but the overhead would be much lower
> than multi-tasking.  Maybe; I haven't thought about
> resource locking.)

Multi-tasking is not the biggest problem, but the logical
sequentiality of most algorithms, access to shared
resources, communication, and load distribution.

For some applications, you can use many processors no
question, but without a revolution in programming languages,
compilers, and such it will be a few carefully selected and
handcrafted pieces of software. (And the so-called
`embracingly parallel problems' like computing a Mandelbrot
set).  But maybe there will be a revolution :-)

Cheers,

Manuel
--------------------------------------------------------------
Next Nomikai: 18 September, 19:30 Tengu TokyoEkiMae 03-3275-3691
Next Meeting: 10 October, Tokyo Station Yaesu central gate 12:30
--------------------------------------------------------------
Sponsor: PHT, makers of TurboLinux http://www.pht.co.jp