grufino
Most performant (and lazy) way to do an operation on 2 lists
Hello!
I am building a quite robust pipeline of operations that performs basically math of all types on thousands (in the future may be millions) of rows.
I would like to know what’s the most performant way to do something that’s been repeating on my code quite a lot, because I always have 2 different lists of variables and need to transform to one by using different types of formulas, so it can become a bottleneck, so a simple example of one case would look like this:
Stream.zip(axs, ays)
|> Stream.map(fn {ax, ay} -> Math.sqrt(ax + ay) end)
Where axs and ays would be lists of float numbers.
By the example you can already understand that it has to be lazy, because we plan to optimize the pipeline using Flow or some other concurrent tool when we have a first version, so advices in that direction are also welcome. 
Thanks in advance!
Most Liked
axelson
I wouldn’t change from operating in one function to a Task based on the code complexity, instead that should be done based on the runtime behavior, for example if a function is parallelizable then it would make sense to use concurrency like a Task (or perhaps GenStage/Flow)
dimitarvp
You are already doing it right IMO.
The only question that’s left is: do you really want to use Elixir for math? Definitely not its forte.
Alternatively, I’d just dump the two collections of numbers in a SQLite database and do the math operation with a SELECT statement. But it depends. If the two lists of data are huge and/or not guaranteed to zip together well then Elixir might be the better tool.
blatyo
The type of math mentioned in the original post seems perfectly fine for Elixir. The implementation of most functions on the math module are NIFs. The stuff that tends to be out of scope for Elixir are heavy math operations like calculating the dot product of two large matrices.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Enabling companies to succeed by building software people love.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"