sezaru

sezaru

Why BEAM nodes try to connect to all other network nodes?

Hello,

Why does the BEAM try to connect each node to all other nodes from the network?

For example, let’s say I have nodes A, B, and C. A needs to talk with B, B needs to talk with A and C only talks with B:

To do this, I configure libcluster with a topology that looks like this:

A <-------> B <-------> C

What I expected was to A and B be connected, and B and C too, but actually A and C connect to each other too.

This is OK in my development environment, but in production, I have more strict firewall rules, so A cannot access C IP and vice versa.

Everything seems to work, but A and C still try to connect to each other, resulting in warnings like this on from my log:

2021-09-25 18:47:47.463 [warn] #PID<0.3141.0> 
↳ global: :"candles@node1.candles.tip-off" failed to connect to :"rocket_dbs@node1.rocket_dbs.tip-off

So, why does this is the default behavior of the BEAM? Can I disable/configure it? What are the advantages/disadvantages of it?

Most Liked

qhwa

qhwa

According to Erlang’s document,

Connections are by default transitive. If a node A connects to node B, and node B has a connection to node C, then node A also tries to connect to node C. This feature can be turned off by using the command-line flag -connect_all false, see the erl(1) manual page in ERTS.

and (as @ityonemo has mentioned)

-connect_all false

If this flag is present, global does not maintain a fully connected network of distributed Erlang nodes, and then global name registration cannot be used; see global(3).

So if you need to discover processes, it will have some issues.

On libcluster’s README, it says:

Features

  • Easy to use provide your own distribution plumbing (i.e. something other than Distributed Erlang), by implementing a small set of callbacks. This allows libcluster to support projects like Partisan.

I haven’t tried Partisan yet, but it works not in “all to all” mode, and it provides some alternatives to the :global registry and process discovery. As I understand one of its selling points is such networking conditions. But maybe you don’t need it since A and C won’t talk to each other?

ityonemo

ityonemo

So one thing that it is important for is to obtain cluster-wide global transactional locks using the :global module. If every node knows every other node, then this is not a problem. If you have a more unusual topology, ensuring that all nodes are aware of the lock is not trivial; I don’t know what guarantees :global makes when you have an unusual topology.

I think if you have a situation where your nodes have a heterogeneous topology you should reconsider using erlang clustering as a “service mesh” or at least look into a different clustering protocol; the original use case for erlang clustering is for symmetrical redundancy, not as a service mesh.

There have been attempts to do so (e.g. “partisan”). I think that project is very interesting but I worry that the abstraction is not quite the right one.

Where Next?

Popular in Questions Top

chokchit
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2733ms. You can configure how long re...
New
nobody
How to bind a phoenix app to a specific ip address? could not find anything about that, nowhere, unfortunately, but for me this is quite...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
RisingFromAshes
I’ve read in another post that it may be possible with a router helper - but I couldn’t find an appropriate one, and tbh, I’m still just ...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
yawaramin
In the Dialyzer docs ( dialyzer — OTP 29.0.2 (dialyzer 6.0.1) ), there is a way to turn off a specific warning for a function: -dialyzer...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29603 241
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 31013 112
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
AstonJ
Please see the new poll here: Which code editor or IDE do you use? (Poll) (2022 Edition) It’s been a while since we first asked this, I...
208 31265 143
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New
svb
Hi! Currently I want to submit a form by pressing the Enter key. However, since my input field is of type “textarea” this is just adds a...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement