@josevalim recently tweeted out:

Btw, we are starting to look into bringing Nx, Livebook, and FLAME to High Performance Computing (HPC). If your company or university works with HPC and you would like to explore these ideas, please do reach out!

Even if you don’t have a lot of experience with HPC, I bet even cursory attempts to survey the scene and publish comparisons with Elixir’s potential would be valuable!

FWIW, completely independently I interviewed some folk working on an HPC project a few months ago, to talk about applications of Elixir in their spaces. They were impressed with Elixir’s capabilities as a resilient, distributed orchestrator of embarrassingly parallelizable work; and intrigued by Nx. Primarily they had open questions about

• Immutability/copy-on-write slowing down node-local performance for computation
• Escape hatches into other languages/tooling that provide highly optimized solutions for their particular domain
• Efficiently and expediently transferring large amounts of data between nodes on successful partial computation to reduce into high-level state

Essentially, these are questions that existing HPC frameworks answer for them, albeit with a lot of fuss and headache in how they write their code, structure their programs, and serialize their data structures. Exploring things like Nx, Rustler, Zigler, and Broadway through this lens could make Elixir more accessible to the domain for future pioneers!

This particular bit is not a concern because we use functional data structures to build a representation of the computation and then compile and execute that instead. So in many cases it can be more efficient, because there is less back and forth between Elixir and the native code doing the work.

Hi @sehHeiden,

I think distributed, remote computation is very much of the minds of the Elixir Data Science community. Check out this Keynote from ElixirConf just a few weeks ago:

I’m sure there are ideas in this area that would be great for a Master’s Thesis.