Maxximiliann
Using concurrency to quickly collate data from multiple sources
Given-
Vehicles is a list of 50 VIN numbers (vin_number).
Colors, Makes, Models, Transmissions, Fuel_economy, Horsepower and Torques are each a list of 50 VIN numbers with their respective features.
For instance:
Colors = [%{vin_number: 5YJSA1DG9DFP14705, color: Black} ...]
Makes = [%{vin_number: 5YJSA1DG9DFP14705, make: DB8 GT} ...]
get_colors, get_makes, get_models, get_transmissions, get_fuel_economy, get_horsepower and get_torques are functions which get the respective colors, makes, models, etc., etc., for a particular vin_number.
Example:
def get_colors(vin_number) do
Enum.find(Colors, &(&1).vin_number == vin_number)
|> Map.get(:color)
end
Collating all of this data into a new map, super_cars, by vin_number-
def super_cars do
Enum.map(vehicles, &
%{
vin: (&1).vin_number,
color: get_colors((&1).vin_number)
make: get_makes((&1).vin_number)
model: get_models((&1).vin_number)
transmission: get_transmissions((&1).vin_number)
fuel_economy: get_fuel_economy((&1).vin_number)
horsepower: get_horsepwer((&1).vin_number)
torque: get_torques((&1).vin_number)
}
)
end
So here’s my question:
How can Task.async be utilized to optimize the time it takes to create the new super_cars map? (Is there perhaps a better approach? Ecto, maybe?)
As always, thanks for your generous and patient insights 
Most Liked
al2o3cr
This isn’t directly relevant to your question about using parallelism, but if you’re concerned about performance consider converting your lists into maps:
colors = [%{vin_number: 5YJSA1DG9DFP14705, color: Black} ...]
map_colors = Map.new(colors, fn c -> {c.vin_number, c} end)
Then a function like get_colors is a map lookup, not a linear search.
chrisjowen
No worries, and although I am not convinced you will need such things for this (I could be wrong just not enough info) I think its worth pointing out your options if you do need to look at parallel processing in Elixir
Firstly all abstractions including Task.async all live on top of the core process model of beam, and its really worth your time understanding this fully.
The next stage is to understand about GenServers and how they encapsulate generic process behaviour (GenServer — Elixir v1.20.2)
After this you may want to still use Task.async or maybe Task — Elixir v1.20.2
If this is not enough for you then the excellent GenStage (GenStage — gen_stage v1.3.2) gives you some real control when producing/consuming large datasets.
Finally, there are interesting abstractions above GenStage such as:
- GitHub - dashbitco/flow: Computational parallel flows on top of GenStage · GitHub
- GitHub - elixir-broadway/broadway: Concurrent and multi-stage data ingestion and data processing with Elixir · GitHub
Basically there are a lot of ways to do concurrent data processing in Elixir
chrisjowen
Maybe, as mentioned there are other overheads in concurrency. If you only have 50 records the question is how many records would each async task process. If you process say 1 item per task it may work out slower than the single process call.
The only way to tell is to try this with different configurations. My gut is that you would be better off keeping this as a single process call and doing smaller optimisations like I mentioned ( @al2o3cr just showed what I mean in their answer, keying by vin number will reduce your lookup time).
Last Post!
Maxximiliann
Thank you gentlemen for your kindly help. It really means a lot especially since I have zero software development background
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Move fast and fix things! Error tracking with less noise for Elixir devs who love to ship. Start your free dev account.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"