Classification of a nodes in a cluster?

I think a lot of k8s people use a service registry like etcd.

I would be interested in another way however as I’m choosing not to go down the k8s route (for now … using Google’s instance groups) but I’m at a place where I need to deregister a node when it receives a sigterm. Perhaps Swarm or Syn.

Is there a clear requirement informed by some other aspect of the design that calls for all of the heterogeneous BEAM nodes to be meshed together in one large pool? Can they be meshed with just their direct peers instead, or in many cases not need to mesh at all?

Most of the time we’d use Pod labels and a ClusterIP or headless Service that selects over those labels to do sibling discovery, which is a model libcluster directly supports. In many environments you don’t have any direct connectivity to etcd that powers the cluster, for security and safety reasons, and some don’t actually use etcd under the hood. IIRC Amazon EKS is actually persisting to DynamoDB or something to that effect.

Honestly they could be meshed only with direct peers. Most of these will only ever interact with one other node type, although it is likely that the api nodes will have to interact with the majority at one point or another.

My purpose in this is mostly to avoid having to write rest api glue between services that could just directly make rpc calls to each other.

I’ve tried to use swarm in the past, but it did not give me useable results. I actually just tried out syn. I decided to use its process groups to register the top level supervisor of each app into the groups :api, and :analytics, then when they start up, their top level pids are registered. Then I can call :syn.get_members(:api) |> Enum.map(&node/1) to get a list of nodes in the system of that type.

I think that works, and I can’t see any reason why it wouldn’t. Does that seem like a good solution?

Are you certain about the libcluster node naming? I’m using libcluster in IoT devices and the node name follows the sname/name given at node startup. Like:

iex(sensor-display@172.31.51.29)1> Node.list
[:“sensor-router@172.31.51.30”]

Not sure how/if there’s an easy way to pass snam/name to the BEAM on k8s from some metadata. Though I think when Node.start is called it can be passed a new name? It’s not too hard to create custom modules for libcluster so you could try that too if you don’t want to use swarm/syn or something else.

Definitely seems reasonable to me!

I’m assuming your IoT devices are using a strategy like gossip or something, which causes each node to send its node name to others periodically, allowing them to connect back.

I think I might go with the syn option. Now if only there were a way to make it log less verbosely.

Yes, it’s gossip. That’s useful to know as I didn’t know the different libcluster modules don’t send the node name.