I’m so glad to have discovered this awesome community. Thanks for creating it! This is my second post, and apologies for any inadvertent breach of protocol by posting this (a variation of a prior reply to a related topic) as a separate post.
I’m just learning Elixir and have encountered a few pronouncements that Elixir is slow and may not be well suited to heavy data/analytics applications. At the same time I’ve seen references to Elixir conference presentations on Big Data applications. As a result I am seeking calibration on the kinds of math-oriented/analytics apps are well suited to Erlang.
A specific use case: I am considering using Elixir/Phoenix for the backend of a Mint-like web app. The app will principally track budget/actual entries, perform predictive / Monte Carlo / scenario modeling and produce datasets for front-end rendered charts.
I’m excited to learn and apply Elixir to this task; but would like to qualify the suitability of Elixir/Phoenix to this type of app.
Elixir, like Java and other systems, complies down to a byte code that runs on a virtual machine. As such low level operations like floating point math usually go through a level or two of indirection. My understanding is that Elixir, Erlang, and their peers may not be the most performant option for number-intensive calculations.
What Elixir and Erlang thrive at is parallelism. In the case of Big Data applications you typically see gains from that kind of parallelism. Running really intensive calculations may not be a forte, but running a bunch of them at the same time could be a win.
I’m not sure if a Mint-like app is considered math intensive or not; but with multiple simultaneous users each viewing scenarios based on on their unique data sets, it’s possible that the app may benefit from Erlang’s multi-threaded/parallel processing capabilities.
I think the math can be characterized as mostly arithmetic and time value of money operations applied to tables/arrays of data up to 1000x1000.
I suspect there is a threshold of math/computation-intensivity where Erlang is not well suited. I’m trying to get a working sense of that threshold, and an appreciation of the classes of apps that fall beyond that threshold.
For something like Mint Elixir is going to be totally fine the number crunching is least intensive of the tasks you will have to perform, polling all the APIs for transaction data and rendering out interfaces will be much higher % of your workload.
Thanks @andre1sk. That’s what I thought; but needed validation absent any calibration of what math/computationally-intensive means in the Elixir context.
Mint.com is a web based budgeting and personal finance tracking application that was acquired by Intuit a few years ago. Mint is the equivalent of Quicken on the web but better and more user friendly.
The target app I’m evaluating Elixr/Phoenix for is not as robust as Mint but does provide basic budgeting, personal finance tracking, analysis and heavy use of graphs to provide guidance.
My thinking is to use Elixr/Phoenix for the backend (database, analytics and preparation of graphing-ready results datasets, API, etc.) plus something like React for the front end (user interaction and reports rendering, etc.).
When we say Elixir may not be suited to do number crunching, we are usually thinking a bit beyond analytics, averages, medians, etc. Because it is not necessarily the mathematical processing that hurts but rather the lack of support for large mutable data structures that would allow us to do implement things like image processing or support vector machines efficiently.
For example, think how you would implement a 100x100 matrix in Elixir and how much copying you would need to do to change a single {x,y} pair using immutable data structures. If you need performance, your best bet is ETS or falling back to C (which is what many languages do anyway).
If you are worried about analytics on something like Mint, Elixir/Phoenix should be fine.
I really appreciate you taking the time to provide such an informative response. I am still learning Elixir syntax and features (before tackling Phoenix) and will add ETS and incorporating C libraries into Elixir/Erlang to my reading list. Can you suggest an Erlang/Elixir finance library or point me in a fruitful direction?
For numerical computing f.e. R enables people to be more productive (higher level language) than C. R itself is written mostly in C iirc. Small sideremark.
Are you agreeing or disagreeing with what Jose said? By “fallback to C” I took Jose to simply mean that most languages end up using C anyway when needing to write optimal number crunching code. R is a good example of precisely that phenomenon.
I only meant that for number crunching a higher level language than C can have benefits. I have a background in a language that started out as a 4GL and later incorporated OO. Number crunching was not it’s strongest point either, they used calls to C or C++ libraries for that also. So I guess I read the same answer as you.
For the rest I don’t understand Jose’s answer completely (but I don’t have a need myself).
Or you may consider calling Julia from Elixir, a relatively new scientific language, close to C in performance.
Form the Julia website: (http://julialang.org/),
<Julia is a high-level, high-performance dynamic programming language for technical computing, with syntax that is familiar to users of other technical computing environments. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. Julia’s Base library, largely written in Julia itself, also integrates mature, best-of-breed open source C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing…>
But not sure how it can be called from Elixir.
Regards.
“Julia is a high-level, high-performance dynamic programming language for technical computing, with syntax that is familiar to users of other technical computing environments. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. Julia’s Base library, largely written in Julia itself, also integrates mature, best-of-breed open source C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing.”
i was recently talking with a trader friend working at one of the bigger banks. His work involves alot of computation-intensive modeling and simulations for options pricing, and he mentioned they have moved to GPU server farms. Pretty much they are using C/C++ with the NVIDIA CUDA runtime API, I am guessing for the reasons Jose Valim mentioned (ease of huge matrix computation, ability to use pointers on mutable data structures, etc).
Here is a brief paper describing the space, if you are interested:
Can someone confirm/deny possibility of calling Julia from Elixir?
I’ve dabbled with Julia awhile, it’s “Elixir-cool”. Can’t say the same for C/C++ (and both Rust and Nim have a lot of library catch-up to do before they have C/C++'s math coverage).
Being able to call Julia would be a great benefit.
But is it possible yet?
You can integrate pieces of code written in other languages in a couple of ways. My first choice is always a port. I’ve blogged a bit about it here. Another options is to run in-process C code through NIF. See here for a good explanation. It looks like Julia can be embedded so both options should be possible.
There is also the scheduling issue with NIF’s. Any function you provide via a NIF needs to run in roughly a millisecond to avoid bolixing the standard scheduler. See the section around “Long running NIF’s” in
This makes embedding potentially very long running math routines difficult. There are a couple ways around this, “dirty schedulers” is the most promising but that is still experimental.
Depending on the exact use case it may make the most sense to use Elixir as a scheduler and use the underlying file system as the data transport. (i.e. build an input file, monitor the math heavy program and read the results in when done.)
Depending on the nature of the code you are using, you can handle scheduling issues without a dedicated scheduler.
You can see an example here of how you can replace a for loop with a recursive function, written in Elixir, which calls the bf_expand NIF. Each call to bf_expand is very short, so there are no scheduling issues.
This not always possible, though, and I’m going to be using dirty schedulers for another app I’m working on.
Can’t say the same for C/C++ (and both Rust and Nim have a lot of library catch-up to do before they have C/C++'s math coverage).
