Questions & Help>Questions

Task.async_stream on all nodes

#task#distributed#stream
sandorbedo

sandorbedo

Task.async_stream on all nodes

Hi,

I use Task.async_stream to execute tasks on an extremely long stream of elements. Basically I use it as a pmap, except that I don’t care about the results, just the side effects of the tasks.

My question is: can I spawn those tasks in the cluster, utilizing all the available nodes? Can Task.Supervisor.async_stream somehow start tasks under more than one supervisors (started one supervisor on all the nodes) in a round robin fashion or select a node by a given hash function?

5 comments
#task#distributed#stream
7 1801 5

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Do you care that the side effects succeed? If you want retries and so on you probably want something like Oban.

4
Post #2
LostKobrakai

LostKobrakai

Do you need cross node orchestration or could all just pull from the same queue? Because for the latter I’d go for a central queue (e.g. sqs) and then run broadway on all your nodes. Broadway does handle error detection/reporting.

2
Post #6

Where Next?

View thread on forum (has 5 responses!)
task
distributed
stream
Home Questions & Help>Questions
#task#distributed#stream
7 1803 5
Last post

Popular in Questions Top

ovidiubadita
Questions & Help>Questions

Elixir for game development
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
#game
30 23559 25
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
#ecto#optimization#delete-method
22 16944 8
New
LegitStack
Questions & Help>Questions

Websocket server in Elixir or phoenix
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
/phoenix#websockets
20 11845 8
New
joeerl
Questions & Help>Questions

Upgrading Elixir - how to check versions, delete, and upgrade?
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
#upgrade
63 41557 13
New
fayddelight
Questions & Help>Questions

Installing elixir via asdf shows zsh: command not found: iex
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
#erlang#asdf
44 16948 17
New
srinivasu
Questions & Help>Questions

Best Practises for Error handling elixir?
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call t...
16 16209 9
New
jason.o
Questions & Help>Questions

No such input `xxxxx` for action ResourceName1.create
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
/ash
3 78595 2
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 20317 15
New
jononomo
Questions & Help>Questions

Websocket connection works on localhost, but get 403 error when deployed via docker
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
/phoenix#channels
8 26815 12
New
lanycrost
Questions & Help>Questions

How To Implement if...else if...else condition
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
#how-to-question
40 52286 6
New

Other popular topics Top

sen
Questions & Help>Questions

How to set environment variables in dev.exs?
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
/phoenix
31 21927 15
New
malloryerik
Learning Resources>Guides/Tuts/Tips/Info

How to use 'phoenix-heex' templates in VSCode
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
/phoenix#vscode#heex
119 17393 34
New
Darmani72
Questions & Help>Questions

Params in the URL and body -- how does Phoenix handle them together?
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
/phoenix#params
17 26931 14
New
gshaw
Questions & Help>Questions

Idiomatic guard clause for checking not nil
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
#guard-clauses
13 43527 3
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? Ecto.Repo — Ecto v3.14.0 has exampl...
#ecto#optimization#delete-method
22 16944 8
New
chrismccord
News>Phoenix News

Phoenix v1.5.0 released!
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
/phoenix#phoenix-release
379 29138 93
New
jay1
Questions & Help>Questions

Why isn’t mnesia the most preferred database for use in Elixir/Phoenix?
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
/phoenix#mnesia
124 21561 22
New
gausby
Chat & Discussions>Discussions

Have you moved away from elixir? If so, why?
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
#discussion#adoption
1207 39297 209
New
Qqwy
Questions & Help>Questions

Updating structs: Map.put vs %Foo{oldfoo | new: value} vs put_in
Original source of discussion: This topic on the Pragmatic Programmers’ Functional Web Development with Elixir, OTP, and Phoenix forum. ...
#maps#structs#pipeline#access
115 28772 31
New
komlanvi
Questions & Help>Questions

LiveView phx-change attribute does not emit event on input text
Hi everyone, I was playing with phoenix liveView but I run into an issue. I have a form and want to validate each input text when the te...
/phoenix#liveview#input-validation
67 20644 33
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Potions

Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement