Pogo - distributed supervisor for Elixir

Pogo is a distributed supervisor for clustered Elixir applications.

It uses battle-tested distributed named process groups (:pg) under the hood to maintain cluster-wide state and coordinate work between local supervisors running on different nodes.

Features of distributed supervisor:

  • automatically chooses a node to locally supervise child process
  • a child process running in the cluster can be started or stopped using any local supervisor
  • ensures a child is started only once in the cluster (as long as its child spec is unique)
  • redistributes children when cluster topology changes

In that aspect it’s similar to Horde or Swarm, but doesn’t provide a distributed registry (Horde and Swarm do). Internals obviously are different - Horde uses ∂-CRDT, Swarm uses Interval Tree Clock for synchronization. Pogo’s local supervisors don’t exchange messages to synchronize state, but rely on Erlang’s process groups, observe their memberships and adjust their local state based on it.

For anyone interested, Pogo’s inner workings have been detailed in an introductory blog post :writing_hand:.

To provide some context, the library was developed at Telnyx as an alternative to Horde as we couldn’t overcome some problems that TBH could have been peculiar to our environment (20+ node cluster with quite dynamic membership). Not that it went smoothly :slight_smile: but since version 0.3 it’s been quite stable.

7 Likes

Thanks for this cool library!

I have several questions:

Does is mean that in case of network split, there will be two instances of the same child_spec: one in the part of network, which is not connected to the leader, and one which is a process, restarted by the topology change? Or will the disconnected from the leader part of the cluster just kill all local children?

In case of a network split there will be two instances of each child process as there is no concept of a leader or majority in pogo. Once the connectivity is restored, extraneous instances will get terminated.

1 Like