saleh-tekm
Is inter-project supervision possible?
Hi everyone,
I’m relatively new to Elixir/Erlang, and I’m working on a demo project. The setup involves a master supervisor, a supervisor, and a worker, and I need to run each of them on separate devices. I’m unsure how these components will communicate when they’re on different machines.
To tackle this, I created separate projects for each component and connected them using Node.connect() and Phoenix.PubSub. However, I realized that this approach doesn’t handle supervision properly, right?
I feel like I might be lacking in the fundamentals of Elixir/Erlang, and I’d really appreciate some guidance on how to implement this correctly.
Thanks in advance! 
Marked As Solved
asabil
This is of course possible, here is a crude example in Erlang:
-module(dsup).
-export([
main/0
]).
-export([
child_init/1,
child_loop/1
]).
-behaviour(supervisor).
-export([
init/1
]).
% Demo entry point
main() ->
{ok, _, Node} = peer:start_link(),
supervisor:start_link({local, ?MODULE}, ?MODULE, [Node]).
% Child process init
child_init(Node) ->
Parent = self(),
{ok, proc_lib:spawn_link(Node, ?MODULE, child_loop, [Parent])}.
% Child process main loop
child_loop(Parent) ->
receive
{echo, From, Message} ->
From ! {echoed, Message},
child_loop(Parent);
{stop, From, Reason} ->
From ! ok,
exit(Reason)
end.
% Supervisor init callback
init([Node]) ->
Flags = #{},
Children = [
#{
id => child,
start => {?MODULE, child_init, [Node]}
}
],
{ok, {Flags, Children}}.
The only thing a supervisor care about is a that a child start function starts a new process, liinks it to the supervisor and return the {ok, Pid} (or {ok, Pid, Info}), the Pid can be started on a remote node as the example above show.
Here is an example session:
> erl -sname node1
Erlang/OTP 27 [erts-15.1.2] [source] [64-bit] [smp:14:14] [ds:14:14:10] [async-threads:1] [jit] [dtrace]
Eshell V15.1.2 (press Ctrl+G to abort, type help(). for help)
(node1@Delta-23)1> c(dsup).
{ok,dsup}
(node1@Delta-23)2> dsup:test().
{ok,<0.101.0>}
(node1@Delta-23)3> supervisor:which_children(dsup).
[{child,<15315.91.0>,worker,[dsup]}]
(node1@Delta-23)4> ChildPid = pid(15315,91,0), ChildPid ! {echo, self(), "hello"}.
{echo,<0.90.0>,"hello"}
(node1@Delta-23)5> flush().
Shell got {echoed,"hello"}
ok
(node1@Delta-23)6> ChildPid ! {stop, self(), crashed}.
{stop,<0.90.0>,crashed}
=SUPERVISOR REPORT==== 19-Dec-2024::18:59:02.505399 ===
supervisor: {local,dsup}
errorContext: child_terminated
reason: crashed
offender: [{pid,<15315.91.0>},
{id,child},
{mfargs,{dsup,child_init,['peer-2242-11228@Delta-23']}},
{restart_type,permanent},
{significant,false},
{shutdown,5000},
{child_type,worker}]
=CRASH REPORT==== 19-Dec-2024::18:59:02.505058 ===
crasher:
initial call: dsup:child_loop/1
pid: <15315.91.0>
registered_name: []
exception exit: crashed
in function dsup:child_loop/1 (dsup.erl, line 36)
ancestors: [dsup,<0.90.0>,<0.89.0>,<0.76.0>,<0.71.0>,<0.75.0>,<0.70.0>,
kernel_sup,<0.47.0>]
message_queue_len: 2
messages: [{exit,crashed},{exit,<0.90.0>,crashed}]
links: [<0.101.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 376
stack_size: 29
reductions: 168
neighbours:
(node1@Delta-23)7> supervisor:which_children(dsup).
[{child,<15315.92.0>,worker,[dsup]}]
As you can see in the last line, the supervisor restarted the process on the remote node.
That being said, you might probably want to take a look at the distributed applications guide for your use case (Distributed Applications — Erlang System Documentation v29.0.2)
Hope this helps.
Also Liked
LostKobrakai
While process links can span across nodes usually supervision and supervision trees are a thing local to a single node.
sbuttgereit
My sense is that a distributed application that is fault tolerant is a different thing from a distributed fault tolerance mechanism. A distributed application that is fault tolerant seems a reasonable and achievable goal (with some limitation), but a distributed fault tolerance mechanism I feel like could never be better than “fragile” or “unreliable” across too many scenarios.
I can see maybe having a specific, maybe more robust, device taking a role as some sort of controller node for commanding distributed processes to start, shutdown (gracefully) as needed, or just to monitor status… but not in the sense of supervision trees, but in the sense that the child processes can receive messages from the central node and that send responses as needed.
For fault tolerance and supervision… I’d be inclined to keep all of that local to the node. To my knowledge it solves a lot of questions distributed supervision might leave you with. What happens if the communication (network) between the child and supervisor isn’t working? Does the child have to detect this and die (self-supervision)? If the supervisor on node a doesn’t see the child in the right time (again, maybe network), does it try to restart the remote child and if the remote child isn’t actually dead is that OK?
I think if I’m really needing something like this, I’m going beyond just baseline OTP, am not really very familiar with OTP, the first thing I’m going to look for is a library (GitHub - elixir-horde/horde: Horde is a distributed Supervisor and Registry backed by Postgres · GitHub, maybe GitHub - bitwalker/libcluster: Automatic cluster formation/healing for Elixir applications · GitHub, etc.) built by others that are better versed in the issues. At least study them to get a handle on it.
I need to disclaim something here: I don’t have much hands-on experience with this in the Elixir/Erlang context so my thinking could well be flawed… but having worked with these kinds of distributed things before I have some instincts for it.
asabil
Does the child have to detect this and die (self-supervision)?
That’s what linking does, the linking is bidirectional, so if the link is severed between the processes, both processes receive an exit signal. What makes supervisors special is that they trap_exit. In this scenario it will mean that the child will terminate if the supervisor is no longer reachable.The supervisor will also receive an exit signal if the child is no longer reachable, but instead of terminating it will attempt to start a new child to replace the old one. Now if that fails many times, the supervisor itself will fail.
This of course doesn’t have to span networks. It could be 2 separate nodes on the same machine, or separate machines connected directly with an Ethernet/serial cable, it really all depends on the topology of the system.
All in all, the most important is to define/understand the failure modes and possibilities of your system. Is there a network? Is the network a possible problem? Is hardware failure a possibility you want to handle? Can someone yank a cable?
Once understood, you can build the right mechanism to handle the failures. No system can handle every single possible failure.
Popular in Questions
Other popular topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"