sandorbedo
Task.async_stream on all nodes
Hi,
I use Task.async_stream to execute tasks on an extremely long stream of elements. Basically I use it as a pmap, except that I don’t care about the results, just the side effects of the tasks.
My question is: can I spawn those tasks in the cluster, utilizing all the available nodes? Can Task.Supervisor.async_stream somehow start tasks under more than one supervisors (started one supervisor on all the nodes) in a round robin fashion or select a node by a given hash function?
Most Liked
benwilson512
Do you care that the side effects succeed? If you want retries and so on you probably want something like Oban.
LostKobrakai
Do you need cross node orchestration or could all just pull from the same queue? Because for the latter I’d go for a central queue (e.g. sqs) and then run broadway on all your nodes. Broadway does handle error detection/reporting.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"