tsukit
I have a GenServer defined as a module X whose each instance can be instantiated with a different parameter. My app creates multiple instances of this GenServer. When serving an coming request, the main flow needs to send a message to each instance of X asynchronously, so each instance can process the call in parallel. The main flow waits to get all the responses before moving on.
I am not sure what’s the best way to do this. Currently this is what I am doing.
defmodule X do
...
def handle_cast({:doit, caller, p1, ..}, state) do
..
send(caller, ...)
{:noreply, state}
end
..
def do_it_async(server, caller, p1, ...) do
GenServer.cast(server, {:doit, caller, p1, ..})
end
end
defmodule Other do
..
def handleRequest(...) do
servers_list = ..
servers_list |> Enum.each(&X.do_it_async(&1, self(), ..)
results = collect_results([], length(servers_list), 0)
..
end
defp collect_results(results, count, received) when count == received, do: {:ok, results}
defp collect_results(results, count, received) do
receive do
{:x_result, ..} ->
collect_results([{..} | results], count, received + 1)
after
5000 -> :timeout
end
end
This is obviously very tedious and error-prone. What is the recommended / common way to do this kind of thing in Elixir?
I am aware of Task.async_stream and I could provide a sync version of the API. However, unless I misunderstood it, I don’t think that’s the right way to do it because it would spawn so many processes just to call the API and wait for the result. Each X instance is already a process.
Where Next?
Trending in Questions
Other Trending Topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #discussion
- #code-sync
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixirconf-us
- #ai
- #blog-post
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #hex
- #security
- #metaprogramming
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Practical resources that improve the lives of professional developers.
Master modern tools, write cleaner code, and tackle real-world challenges.
Producing high quality Elixir screencasts since 2017.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 5- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
pmangalakader
Hi @tsukit,
If your intention is just parallel processing or computation along with throttling, then you can take a look at Elixir Flow.
If it is something different, then a little more elaborate would help understand the problem you’re trying to solve.
Thanks.
entone
Registry also provides a nice way to do pub-sub without having to introduce GenStage or back pressure or anything like that, Registry — Elixir v1.20.2
tsukit
Thanks @pmangalakader. It’s for parallel processing. I need to do task T1, T2, .. Tn in parallel. I create different instances of X for this purpose and also to prevent having to spawning so many process for each request. I’m not sure how this will be model with Flow though (I am very familiar with stream concepts like Akka Stream). Would you be kind giving an example using Flow?
bottlenecked
Hi, you should probably go with what @entone suggested.
Simply use the Registry to register each instance of your GenServer process and you’re good to go
pmangalakader
@tsukit, the example in the link can easily illustrate word count in parallel. I had the problem of loading GeoData into a database before starting an application. I had tried to use GenServers with poolboy, but it required a lot of manual tweaks to get it to optimized working. So I tried to search for alternatives and found
Flowthat had used Genstages and sufficed my needs at that time. At the moment I don’t have that code with me.The below thread has some important needs for understanding Flow:
The magic here is the
:min_demandand:max_demand, it uses the number of cores by default, but you can increase it or decrease it based on your needs.