Is Elixir used on supercomputers for high-performance computing (HPC)

Questions / Help
, , , ,
1

High-performance computing (HPC) is the use of supercomputers and parallel computing techniques to solve complex computational problems.

There exists a similar question from June 2016: Erlang/Elixir and supercomputers

I was wondering if there are use cases where Elixir was used for high-performance computing (HPC) on supercomputers, especially due to the years that have gone by since 2016 and the recent developments (Nx, ML and AI use, Livebook, etc.).

Thank you in advance!

1 Like
2

It has definitely changed since 2016 because of Nx. It depends on the type of work/algorithms you want to apply, but with Nx you can do numerical computation efficiently on individual nodes (especially if they also have a GPU) and thanks to OTP you can easily communicate between the nodes to coordinate the work. We did that to run some evolutionary algorithms using HPC, published in https://dl.acm.org/doi/10.1145/3512290.3528753 : )

3 Likes
3

Hi Jonatan,

Thank you very much for your reply!

The paper you are mentioning sounds very interesting, I will try to get my hands on it.

Which supercomputer did you run your HPC code on, if it is a known one: Supercomputing in Europe

Do you know of any other publications, articles or books that have content on Elixir and HPC?

Cheers Patrick

4

Which supercomputer did you run your HPC code on

Prometheus (Cracow, Poland) : )

Do you know of any other publications, articles or books that have content on Elixir and HPC?

Nothing in particular, especially related to Nx. I think there are a few papers that used Erlang and HPC, like this one, you can check out the author to see if there’s anything relevant for you.

1 Like
5

Thank you very much :pray:t2:

6

Dear @josevalim,

@jonatanklosko suggested asking you about the following questions:

Are there any new developments, papers, or use cases for Elixir on supercomputers for HPC?

On another note, are there any open research-related topics (e.g. Master thesis or PhD topics) in the Elixir space?

Thank you very much in advance.

Cheers Patrick

7

Hi Patrick, the most recent effort that I am aware of is FLAME Slurm: GitHub - marcnnn/flame_slurm_backend: A FLAME Backend for Slurm.

I have expressed interest for running the Erlang distribution over RDMA, and if you are either a student or a supervisor interested in taking this forward, you can reach out to me.

Finally, we are preparing to implement GPU-to-GPU operations directly in Nx.

2 Likes