Questions & Help>Questions

Task.async_stream on all nodes

#task#distributed#stream
sandorbedo

sandorbedo

Task.async_stream on all nodes

Hi,

I use Task.async_stream to execute tasks on an extremely long stream of elements. Basically I use it as a pmap, except that I don’t care about the results, just the side effects of the tasks.

My question is: can I spawn those tasks in the cluster, utilizing all the available nodes? Can Task.Supervisor.async_stream somehow start tasks under more than one supervisors (started one supervisor on all the nodes) in a round robin fashion or select a node by a given hash function?

5 comments
#task#distributed#stream
7 1801 5

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Do you care that the side effects succeed? If you want retries and so on you probably want something like Oban.

4
Post #2
LostKobrakai

LostKobrakai

Do you need cross node orchestration or could all just pull from the same queue? Because for the latter I’d go for a central queue (e.g. sqs) and then run broadway on all your nodes. Broadway does handle error detection/reporting.

2
Post #6

Where Next?

View thread on forum (has 5 responses!)
task
distributed
stream
Home Questions & Help>Questions
#task#distributed#stream
7 1803 5
Last post

Popular in Questions Top

marius95
Questions & Help>Questions

Javascript Event Handler is firing twice
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
/phoenix
0 15200 6
New
skosch
Questions & Help>Questions

Put/update deep inside nested maps (and auto-create intermediate keys)
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
#data-structures#maps#immutability
52 20278 11
New
ovidiubadita
Questions & Help>Questions

Elixir for game development
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
#game
30 23559 25
New
johnnyicon
Questions & Help>Questions

How do I use the Postgres JSONB / Postgrex JSON extension?
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
#ecto
22 17673 9
New
earth10
Questions & Help>Questions

Understanding Elixir/Phoenix performance
Hi, I’m just starting to build a side-project with Elixir and Phoenix and doing some basic test with Elixir alone. What strikes me is th...
/phoenix#performance#concurrency#raw-speed
222 19344 43
New
stefanchrobot
Questions & Help>Questions

How to decode a JSON into a struct safely?
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
#structs#json
29 21152 26
New
vegabook
Questions & Help>Questions

How to serve an img with Phoenix?
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
/phoenix
15 18745 5
New
vonH
Questions & Help>Questions

How do I kill a process ` #PID<0.186.0` in iex?
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
#iex#processes#how-to-question
37 21444 8
New
itssasanka
Questions & Help>Questions

Why use `utc_datetime` over `naive_datetime` for Ecto?
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: The documentation above suggests that while ...
#timezones
115 15182 19
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 20317 15
New

Other popular topics Top

sen
Questions & Help>Questions

How to set environment variables in dev.exs?
Hi All, I set a environment variables in dev.exs , like below code. when i start server, how can i set the ${enable} value? thanks. d...
/phoenix
31 21927 15
New
Darmani72
Questions & Help>Questions

Params in the URL and body -- how does Phoenix handle them together?
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
/phoenix#params
17 26931 14
New
Nvim
Chat & Discussions>Discussions

Django vs Phoenix
Anybody knows a comprehensive comparison of Django and Phoenix, thanks for the help. Where are they similar? Where do they differ the m...
/phoenix#learning-elixir
184 21696 82
New
greenz1
Questions & Help>Questions

Best way to send multiple files as HTTP response
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
/phoenix#api
3 18924 3
New
JeremM34
Questions & Help>Questions

How can I check Phoenix version?
Hello, how can I check the Phoenix version ? Thanks !
/phoenix
35 28221 8
New
johnnyicon
Questions & Help>Questions

How do I use the Postgres JSONB / Postgrex JSON extension?
Hi all, I’ve just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I’m trying to use Postgres...
#ecto
22 17673 9
New
minhajuddin
Questions & Help>Questions

Using List.first instead of Enum.at(0)
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
#code-style
76 33670 14
New
jerry
Questions & Help>Questions

Ecto query using like/ilike in query
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
#ecto
17 16876 10
New
Emily
Questions & Help>Questions

How To Get Phoenix & VueJS working Together?
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
/phoenix
93 22614 42
New
Brian
Questions & Help>Questions

Import a module from a file into IEX
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
#iex
35 32626 16
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

AppSignal

Catch errors, track performance, monitor hosts and more.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Popular Tags

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement