Alex66

Alex66

How best to approach a live migration of pipeline segments across BEAM nodes?

Running an FBP pipeline where each node is a GenServer. Need to move a segment of the running pipeline (multiple connected nodes) from one BEAM node to another without losing messages or breaking the flow.

Example: nodes A → B → C → D running on server 1. Want to move B → C to server 2 while pipeline is processing.

Questions:

  1. Drain and restart, or something smarter?

  2. How to handle in-flight messages during migration?

  3. Coordinate the cutover so upstream (A) starts sending to new location?

  4. Is there a pattern for this, or just stop/migrate/restart?

Using Pulsar/Artemis as message broker, so could potentially buffer there during migration. But curious about pure BEAM approaches too (mailbox draining, etc).

Most Liked

allen-munsch

allen-munsch

I haven’t done this using FBP, or BEAM. So I am drawing my answer from my experience in just database migrations ( postgres ) from one cloud to another with near zero downtime ( very little buffered draining )

The way I have done something similar in the past was via WAL streaming both with a timed syncronization window where queries timed out for a small bit while the buffers drained over.

Proxy stream original B → C to the 2nd B → C

verify 2nd B matches 1st B, then stop sending to 1st B

Kinda like: Blue–green deployment - Wikipedia

Someone told me once it’s a combination of the 2, a Cyan deployment.

If it’s okay to take it offline for maintenance for an hour or 2, that’d be easiest, just imo

manhvu

manhvu

In this case, you can use hot code reloading for the GenServer , update its state in the code_change/3 callback, and route requests to the new node.

Alex66

Alex66

Thanks everyone for the input! Wanted to share what I ended up implementing.

Already had:

  • Remote Output / Remote Input nodes for cross-BEAM communication

  • Circuit breaker with message buffering when target unavailable

  • Auto-recovery when connection restored

What I added:

  • Runtime pipeline modification (add/remove nodes while running)

  • Runtime property injection (change node config without restart)

  • Dynamic connection changes (rewire targets mid-flow)

The migration approach:

  1. Circuit break at the split point (Remote Output buffers messages)

  2. Send pipeline segment config (JSON) to target server

  3. Start the segment on new server

  4. Update Remote Output’s target channel at runtime

  5. Buffer drains to new location

Nodes are stateless transformations — state lives in the message flow. So “hot migration” is just runtime topology modification.

Appreciate the discussion!

Where Next?

Popular in Questions Top

joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
hariharasudhan94
lets say i have a sample like a = 20; b = 10; if (a > b) do {:ok, "a"} end if (a < b) do {:ok, b} end if (a == b) do {:ok, "equa...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
shijith.k
I am trying to start a new phoenix project with elixir 1.9, but mix phx.new does not work. It says that ** (Mix) The task "phx.new" could...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New

Other popular topics Top

electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
AstonJ
Posting this to see if we can make things easier for people to get into Neovim. If you use Neovim and have a favourite distro please let ...
New

We're in Beta

About us Mission Statement