fireproofsocks

fireproofsocks

While exploring with Oban I was made aware of some of the risks involved with serialization, in particular with serializing anonymous functions (see Risks of serializing structs and captured function). The related/belated follow-up thought that occurred to me was “is it safe to send streams to separate nodes for execution?” Streams are essentially captured functions, right? So isn’t there a risk that if the stream is created on one node and then sent to another node for execution, then the stream might fail to execute because maybe the receiving node doesn’t have the same modules available or maybe it has been updated slightly causing the function identifier to change? Or does this issue only come up if the thing is serialized?

My specific use case is dealing with data processing – in a nutshell, one process prepares streams and sends them to another process where they are run with specific boundaries for concurrency and rate-limiting enforced. It seems to work fine when running on a single node, but I wanted a reality check… could this all fall apart if 2 nodes were involved or if a hot-code update were applied?

Thanks for any thoughts on the matter.

Showing Posts 1 to 8

LostKobrakai

LostKobrakai

They can be, but they don’t need to be. Any Enumerable.t can be “a stream” and be used with Stream API. Given your other explanations I’d expect the streams you build to be a bunch of callbacks though.

al2o3cr

al2o3cr

I’m not sure you can trust this boundary - the captured functions aren’t IMO the biggest risk, it’s what the functions could reference that might not make the transition. For instance, things like ports or local PIDs won’t work.

fireproofsocks

fireproofsocks OP

This is helpful, thank you.

When there are multiple nodes involved, is there a shared registry or something so that any node could look up a PID or a port?

Maybe related… would it be possible to write a function that could inspect a value to see whether or not it’s safe to ship across process boundaries? E.g. something that could detect ports/PIDs? (actually… maybe something as simple as doing JSON encoding would probably be a decent check on this). Or is this going against the grain?

LostKobrakai

LostKobrakai

Why “build” those streams on a different node in the first place? I’m not sure I see why you’re doing that.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

My general advice for this sort of thing is to use inter-node communication to coordinate work, but not do the work, if that makes sense. Each node should do its own work on the data it fetches, and not try to recruit other other nodes to help with that within a given “unit” of work. If your overall processing pipeline has logical steps or chunks within it that produce artifacts then you can sometimes have a node work on stuff up and until a chunk is done, and then the next “step” could go in a queue or similar that gets run on another node. But within a given step it’s going to be both easier and generally faster to just do the work in one node.

hst337

hst337

Yes, there is a risk. If you send started but unfinished stream on the other node, this stream will start from the beginning.

There is no general rule, streams are unsafe even when they’re not distributed (when you exit the stream during traversal with something like throw which very common and even Elixir core team does this).

So I’d suggest using some other primitive like process. For example, you can create some proxy process with support for migration from node to node, which proxies GenStage producer

fireproofsocks

fireproofsocks OP

good question. The short answer for one use case is to adhere to DRY principles: e.g. wanting to reuse GenStage components that do the processing, especially when the processing needs to happen with shared rate-limiting/concurrency limits. E.g. stream X does one set of API operations, stream Y does another set of operations against the same API, and they get processed in the same GenStage consumer because all operations share the same usage limit defined by that API.

It sounds like if the execution flow is expected to be distributed across multiple nodes, then perhaps it’s better to send messages with “simple” payloads, e.g. a simple list instead of a stream.

LostKobrakai

LostKobrakai

Yeah, sending data to the computation is simpler than sending the computation around.

— All posts loaded —

Where Next? Top

Trending in Discussions Top

AstonJ
The obligatory hello world thread! Who are you and where are you from? :stuck_out_tongue:
4663 61218 595
New
cblavier
Hey there, It’s been more than a year since we started using LiveView as our main UI library and building a whole library of UI componen...
515 12931 101
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
sntran
In my third year of college, around 2010, I found Erlang. A few years later, I found Elixir. I liked the ideas at once: many small proces...
New
gtcode
I’ve been tinkering with Lynx (by José) via hold_ledger. I used AI to write the code, a separate specification, the tests, and the Lean p...
New
AstonJ
A remark @Garrison made… ..has already inspired a thread for the technical side of reclaiming our internet, but there is of course anot...
New
ManuelGarcia
Hi everyone! I’m the developer of PaperForge, and I’ve been working on something alongside it that I’d like your opinion on. I’m conside...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews