CherryPoppins
So im looking to do something like below where I get a list of posts from the db and then run an api call on each post's comments and i want it to all happen as concurrently as possible. Is nesting task bad practice?
defmodule MyApp.ObanWorker do
def perform do
posts = MyApp.Repo.all(Posts)
MyTaskSupervisor
|> Task.Supervisor.async_stream(posts, &MyApp.ApiClient.do_it/1)
|> List.flatten()
|> Enum.each(fn {_, message} -> Logger.info(message) end)
:ok
end
end
defmodule MyApp.ApiClient do
def do_it(%{comments: comments} = post) do
MyTaskSupervisor
|> Task.Supervisor.async_stream(comments, &make_call/1)
|> Enum.filter(&(elem(&1, 0) == :error))
end
def make_call(comment) do
result = # ...make api request
case result do
{:ok, %{status: 200}} -> {:ok, "good job."}
_ -> {:error, "not good for #{comment}."}
end
end
end
Trending in Questions
Hello everyone!
I am new to coding with Elixir/Erlang. And i am looking for some utilities for fuzzing testing like AFL++.
Does anyone ...
New
I’m very new to Elixir, so sorry if this is a silly question:
I am wondering what happens to the mailbox of a process when the machine i...
New
How do I tell Hologram not to escape HTML when substituting into templates?
{Textile.render(@product.long_description)}
escapes the HTM...
New
Background
When trying to execute mix release on a Windows 11 machine for a Phoenix project I get the following error:
* assembling mark...
New
I load tested a plain Phoenix chat app backed by Postgres with 100 million messages in it, everything on default settings, to see what br...
New
I’d like to create a file with 600 permissions (rw-------), but there’s a race condition if I create it and then change the permissions. ...
New
Is there a way to import record definitions in the iex shell for pretty printing (similar to the rr command in the erl shell)? This woul...
New
Other Trending Topics
Senior Full Stack Engineer (Elixir + React) - Mux, Remote USA
Introductory paragraph
Mux is video infrastructure for developers and their...
New
Hey there,
I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first.
...
New
Hi everyone :waving_hand:
I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly)
AI dream has many faces, but general...
New
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #elixirconf-us
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #security
- #elixirconf-eu
- #processes
- #api
- #forms
- #hex










Showing Posts 1 to 3- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
al2o3cr
One thing to watch out for with tasks is copying large data structures - in this case,
commentspresumably has a lot of entries (thus the need for concurrency) so copying it is expensive.gregvaughn
I wouldn’t nest them. That would increase the complexity of tuning via max_concurrency. I would start with a list of comments that belong to the desired posts in the original query.
dimitarvp
Don’t nest them because you’re likely dealing with increased copying of values and with parallelism guarantees that will no longer hold true (i.e. originally you made the code to do no more than 20 tasks in parallel but with nesting that can balloon further).
Alternative thing you can do is to separate all the tasks and queue them one by one through, say, commanding a
GenServerto pull them one by one from a queue, and then each task can useTask.async_stream(with only one level and no nesting).