tmbb

SciEx - scientific programming for Elixir (based on bindings to rust's ndarray)

SciEx - Scientific programming

Code here (very early stage): GitHub - tmbb/sci_ex: Scientific programming for Elixir · GitHub

I have decided to start a small project with the goal of giving real scientific computing capabilities to Elixir. The plan is to implement something like NumPy and SciPy. I plan on basing everything on bindings to the rust library ndarray and probabbly others like faer. The focus will be on “classical” scientific computing. Machine learning tools will not be on the forefront here. I won’t be using Nx because I personally don’t like it much and can’t even get it to run in my personal laptop.

The following IEx session shows what’s possible (sorry for the funky output, I’m currently reusing the normal debug functions for ndarray’s types instead of something more “elixiry”):

iex(1)> alias SciEx.Array1
SciEx.Array1
iex(2)> Array1.zeros(100)
#SciEx.Array1<[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], shape=[100], strides=[1], layout=CFcf (0xf), const ndim=1>
iex(3)> alias SciEx.Array2
SciEx.Array2
iex(4)> Array2.zeros(10, 10)
#SciEx.Array2<[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]], shape=[10, 10], strides=[10, 1], layout=Cc (0x5), const ndim=2>
iex(5)> arr2 = Array2.zeros(10, 10)
#SciEx.Array2<[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]], shape=[10, 10], strides=[10, 1], layout=Cc (0x5), const ndim=2>
iex(6)> SciEx.cos(arr2)
#SciEx.Array2<[[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
 [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]], shape=[10, 10], strides=[10, 1], layout=Cc (0x5), const ndim=2>

16 comments

#sci_hub #scientific #library-rfc

8 805 16

2026-03-10 11:14:29 UTC

Most Liked

tmbb

Added a lot of functionality today. SciEx now support arrays of floats of dimension up to 6 (SciEx.Float64.Array1, …, SciEx.Float64.Array6). Most of the rust code (and some of the elixir code) in the bindings is now programmatically generated from the Elixir code as part of a separate build step.

I have added support for the most common floating point functions, including some special functions. With the new programmatic code generation tools, adding support for more functions is trivial. The functions are currently only vectorized in the first argument, but I plan on adding vectorization on the other arguments.

I have added support for drawing random variables according to the given distribution. For example, one can draw 15 values from a Cauchy distribution:

iex> SciEx.Random.draw_from_cauchy(0.0, 1.0, 15)
#SciEx.F64.Array1<[0.5075428433765116, 6.5273500168873, -0.09238828295381869,
-0.5017462511724131, 0.31446532306974234, -0.12084509091920821, -0.35170244950655,
-7.796726646392933, 0.4049382192392038, -1.5996980781784342, -1.0434302700931213, 
0.8270864602500656, -1.7333229127273764, 0.541653889771595, 129.4432667174581],
shape=[15], strides=[1], layout=CFcf (0xf), const ndim=1>

The PDF and CDF for the supported distributions is also implemented. However, they are only vectorized on the first argument.

I’m not sure on whether I’m taking the right approach regarding the rust code, though. I am writing (and compiling) separate rust functions for each supported type combination because that makes the interop with Elixir easier, although I could probably get away with generating less rust code…

I have also added support for arithmetic operators on arrays and scalars. Array types and dimensions must match, otherwise SciEx will raise a mistake. Everything is very explicit (you can use SciEx.Operators).

iex(4)> a = SciEx.Random.draw_from_normal(0.0, 1.0, 10)
#SciEx.F64.Array1<[0.980546308591242, -0.4124865578379766, -0.2678229458353892, 0.13895119159920113, -0.05895938590440134, 0.5463383504051798, -0.47139019152800626, 1.1432875805108502, -1.053279376911445, -0.4890897953025381], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>
iex(5)> b = SciEx.Random.draw_from_normal(0.0, 1.0, 10)
#SciEx.F64.Array1<[-0.6112979257481231, 0.04187616378275234, 0.024658043411414837, -0.7423086166282216, 2.5287898587121993, -1.5968438074887625, 0.3028680311098233, 1.1657682958696634, -1.3245816353940532, 1.1693983808728805], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>
iex(6)> 0.5 * a + 0.7 * b # This will fail beause it will try to use the Kernel operators
** (ArithmeticError) bad argument in arithmetic expression: 0.5 * #SciEx.F64.Array1<[0.980546308591242, -0.4124865578379766, -0.2678229458353892, 0.13895119159920113, -0.05895938590440134, 0.5463383504051798, -0.47139019152800626, 1.1432875805108502, -1.053279376911445, -0.4890897953025381], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>
    :erlang.*(0.5, #SciEx.F64.Array1<[0.980546308591242, -0.4124865578379766, -0.2678229458353892, 0.13895119159920113, -0.05895938590440134, 0.5463383504051798, -0.47139019152800626, 1.1432875805108502, -1.053279376911445, -0.4890897953025381], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>)
iex(6)> use SciEx.Operators # this imports the operators in SciEx and hides the ones in the Kernel
SciEx
iex(7)> 0.5 * a + 0.7 * b  
#SciEx.F64.Array1<[0.06236460627193485, -0.17692996427106167, -0.1166508425297042, -0.45014043584015456, 1.7406732081463385, -0.8446214900395438, -0.02368747398712684, 1.3876815973641894, -1.4538468332315597, 0.5740339689597472], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>

Post #2

tmbb

Due to the fact that Explorer (the elixir library) is based on Polars (the rust library), which is based on a rust implementation of Arrow (a standard to store data in memory), I have tried to re-implement SciEx to be based on Arrow instead of ndarray, in order to make it comaptible with Explorer. It would also make it much easier to store and move data around.

However, I haven’t been able to make it work in a satisfying way… Arrow doesn’t even provide a multidimensional array type you can operate on as a unit. It does provide tensors, but it does not provide vectoried operations on tensors and I can’t think of a good way to implement something like a Fast Fourier Transform algorithm on Arrow datatypes.

If anyone can think of a way to “port” SciEx to Arrrow I’d be very interested.

Post #12

tmbb

I have added support for loading 2D-arrays from grayscale PNG images and support for contouring “heightmaps” (represented as 2D arrays) using the marching-squares algorithm. @kip I’m sorry, but I haven’t been able to work on integrating SciEx with your image processing tools yet.

I plan on wrapping my own kernel density estimation (KDE) functions and expose them to Elixir as part of SciEx. Once this is done, I plan on using SciEx for the most computationally heavy parts of my Quartz plotting library.

Post #11

Last Post!

tmbb

A release has been published on hex, because I’ve found myself using parts of it in other projects. A lot of the functionality still doesn’t have proper tests, and if you’re not willing to read the code yourself, everything that isn’t simple arithmetic or function calls is probably not yet good enough for production use.

I have added new features, including support for hierarchical clustering based on dissimilarity matrices. This can be useful to perform clustering on sentence embeddings from LLMs.

I have added support for 1D kernel density estimates and 2D contouring (isolines). This package doesn’t plot them and never will, but it can be integrated with packages for plotting.

I have also added utilities to generate multidimensional arrays from Elixir lists. Thus is not very useful for practical use but it is useful for testing and exploring small examples.

In case engine wants to contribute, I suggest the following topics:

Adding tests to the clustering logic (maybe take them from SciPy, who’s API I’m trying to emulate)
Adding support for RustlerPrecompiled
Find a way of integrating tests suites from other languages for the functions (even though the functions are taken from the rust stdlib or some other rust package, testing the implementation over the arrays seems like a good idea)

Post #17