zenchild

Announcing libcluster_dynamic_srv: Dynamic Port Clustering for Elixir

I’ve been working on a way to cluster Elixir nodes in dynamic environments without needing to set static ports. This work was heavily inspired by the Caravan package, but differs in one key way: you don’t need to specify a port in the node name. The port can be completely dynamic and is discovered using DNS SRV records. It also has a libcluster DNS strategy for clustering with the SRV records. It is very similar to the K8s strategy in the core libcluster library, but is a bit more flexible around service naming.

I’m currently using this with Nomad and Consul, and I’ve also created a demo lab that uses Docker + Consul if you want to play around with it.

How It Works

Unlike traditional Elixir clustering approaches that require fixed ports, this library leverages DNS SRV records to discover both the host and port information for each node. This makes it perfect for containerized environments where ports are often dynamically assigned.

Future Possibilities

This approach should also work with Kubernetes, though I’m not as experienced with K8s. If someone is interested in exploring this integration, I’d love any feedback on the approach or contributions to the examples folder!

Cheers,
Dan Wanek

Showing Posts 1 to 4

Show Best Posts
Show All (oldest first)
Show All (newest first)

zenchild

Moving to 1.0.0

We’ve been using this library for awhile now and things have been stable so I’ve decided to move it to 1.0.0. This version does have some clean-up and fixes a DNS polling issue we discovered, but it is backward compatible with pre-1.x configurations.

Changelog

Fixed

Replace double poll scheduling with handle_continue/2 to ensure a single, clean poll chain
Use handle_continue(:poll, state) for initial poll instead of calling do_poll/1 directly in init/1
Add catch-all clause in get_nodes/1 to handle unexpected resolver responses without crashing
Fix regex character class [a-z0-9-_] to [a-z0-9_-] to avoid ambiguous range matching

Added

Test coverage for unexpected resolver responses
Test coverage for SRV records with non-matching hostname formats
Test coverage for missing :service config
Test coverage for meta: nil initialization path
Test coverage for self-exclusion from the node list
Test coverage for recurring polls after the initial poll

Version 1.1.0 Released

We’ve recently added a few things to make libcluster-dynamic-srv work a little more consistently in production environments. Some notable things from the changelog are listed below.

Changelog

Added

disconnect_on_deregister config option (default true). When set to false, nodes that disappear from DNS but still have a live distribution connection are left connected and only removed once the connection drops naturally. This prevents spurious :global partition-prevention cascades during graceful shutdowns that use a deregistration delay.

Fixed

DynamicSrv.Epmd.address_please/3 now returns {:error, reason} instead of crashing when DNS resolution fails (e.g. :nxdomain). The distribution layer handles this as a failed connection attempt rather than a process crash.
Enable EDNS(0) (edns: 0) on all :inet_res.lookup/4 calls in both the clustering strategy and the EPMD module. Many modern DNS resolvers (e.g. Consul, CoreDNS) require EDNS(0) to return full SRV responses over UDP; without it, lookups could silently return empty or truncated results and cause cluster discovery failures.

Post #2

tpendragon

Thank you so much for this! This has really simplified our infrastructure: Use Consul for Elixir Clustering by tpendragon · Pull Request #1358 · pulibrary/dpul-collections · GitHub.

Post #3

zenchild

I’m super glad you’re finding it useful. Let me know if you run into any hiccups. We’re using it for two services and it’s been great for us, but always curious to hear what others run into.

Post #4

— All posts loaded —