elbasti

elbasti

Waiting for multiple tasks in a genserver

I have a genserver that wants to spawn multiple tasks (in this case multiple api calls to different services) and wait for all of them to return, and then process all of the responses simultaneously.

I was originally going to use Task.await_many/2 to well, wait for all of the tasks to return, but this comment in the documentation made me doubt:

It is not recommended to await long-running tasks inside an OTP behaviour such as GenServer

The recommendation is to instead match on the message coming from a task inside your GenServer.handle_info/2

The docs have a really nice example for monitoring a single task, but how would you write that handle_info function to wait for multiple tasks to return (especially things like handling timeouts for the individual tasks)?

Most Liked

vfsoraki

vfsoraki

To build upon the proposed solution, I’d say you can batch :do_all_the_things in one separate process. Ignoring task supervisor, this is a rough code.

def handle_call(:do_all, _from, state) do
urls = […]

gen_server_pid = self()

batch_pid = spawn(fn ->
  tasks = Enum.map(urls, START_TASK_FOR_ONE_URL)
  result = Task.await_many(tasks)
  # You can include a kind of ID here too to distinguish batches
  send(gen_server_pid, {:batch_result, result})
end)

# Keep batch_pid if you need it
{:reply, :ok, update_state(state, batch_pid)}
end

def handle_info({:batch_result, result}, state) do
  # Process results
  {:noreply, state}
end

You get the idea. You start a separate process that spawns processes to process each mini task, then gathers all results and sends them to gen server in one message.

al2o3cr

al2o3cr

What’s responsible for deciding to make those requests? If it’s a request from outside doing something like GenServer.call(pid, :make_all_the_requests), what should happen if another make_all_requests comes in while the first one is still processing?

One straightforward alternative would be to have the GenServer launch a single Task which then launches the others and does an await_all.

stevensonmt

stevensonmt

I would assume Task.Supervisor.async_stream_nolink is the starting point. Interesting question and I hope someone can provide a concrete example for you.

Where Next?

Popular in Questions Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list. ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
dokuzbir
I want to highlight html closing tags when i click a html tag. That works in .html files but doesnt work for html.eex templates. How can...
New
sen
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 130286 1222
New
nobody
Hi! In PHP: $_SERVER[‘SERVER_ADDR’] - in Elixir? Searched the docs for ip address and the web, no good results. Thanks!
New
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement