eDev

eDev

Erlang/Elixir and supercomputers

This guy didn’t get the memo about Erlang!

“As a computer scientist it’s difficult writing software that can take advantage of and control large numbers of computer cores,” said Professor Les Carr from the University of Southampton.

"This is why supercomputers are restricted to specialised applications - you need very specialised computing needs to take advantage of them.

In all seriousness - would BEAM be a good fit for super computers like these?

Most Liked

bbense

bbense

I started out my career working at the San Diego Supercomputer Center on some of the very early multiprocessor supercomputers of the day (i.e. when a 1000 cores was a supercomputer). My interest in concurrency started there and eventually lead me to Erlang and Elixir.

While there are some simulations that could effectively use the BEAM, most supercomputer type problems don’t translate well to the actor model due to the connectivity of the data. For large physical simulations, every data point affects the value of every other data point in the simulation.

The Actor model assumes a network of minimal connections. If every process needs to talk to every other process then communication dominates costs as you scale the problem.

Now on the other hand, almost all of the science of numerical computation is figuring out when it’s “okay” to ignore the connections between data in a problem and partition the problem into smaller solvable problems.

I think the BEAM could work in these environments as scheduler and coordinator, but you’d need a specialized core of mathematical routines that currently doesn’t exist.

uranther

uranther

These supercomputers are working on parallelizable problems, yes, but need raw speed such as for modeling climate change. They might even use specialized languages for the computations. A BEAM language may be good for supervising the process farm but other than that there needs to be mutable data structures and fast matrix computations.

Qqwy

Qqwy

TypeCheck Core Team

I really do wonder what would happen if we had a pure dependently typed actor-model functional language. This would enable us to use crazy tricks like process-based memoization, and have the compiler do all kind of weird things for us to make programs faster.

It is completely possible to use mutable data structures in a purely functional language by using a State Monad Transformer (But I have to admit, this is the kind of stuff that Mathematics Professors discuss about with squiggely lines and vague terms – I don’t understand how this works myself either right now).

Last Post!

pat_rick

pat_rick

This question was raised in June 2016. Would the answers to this question today (October 2023) be different than they were back then?

Where Next?

Popular in Discussions Top

AstonJ
Are there any Elixir or Erlang libraries that help with this? I’ve been thinking how streaming services like twitch have exploded recentl...
New
PragTob
Hello everyone, I know we had quite some threads (read through lots of them) about background job processing but it remains a hotly deba...
New
tomekowal
Hey guys! I want to create a toy project that shows a chart of temperature over time and updates every 5 seconds. I feel LiveView is per...
New
Fl4m3Ph03n1x
Background This question comes mainly from my ignorance. Today is Black Friday, one of my favorite days of the year to buy books. One boo...
New
cvkmohan
The upcoming Phoenix 1.6 release looks very interesting. Became a habit to watch the commits - and - what they are bringing in. phx.gen...
New
paulanthonywilson
I like Umbrella projects and pretty much always use them for personal Elixir stuff, especially Nerves things. But I don’t think this is ...
New
shishini
I think this twitter post and youtube video didn’t get as much attention as I hoped I am still new to Elixir, so can’t really judge ...
New

Other popular topics Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New

We're in Beta

About us Mission Statement