Connect nodes using Libcluster with docker-compose

I’m trying to use libcluster in order to connect elixir nodes. You can see here my simplified version of docker-compose.

 version: "3.7"
 services:
   application_1:
     build:
       context: application_1/.
     ports:
       - "4321:4000"
   application_2:
     build:
       context: application_2/.
     ports:
       - "1234:4000"

Everything is starting perfecly, each services is working fine. When I’m connected to application_1, I’m able to ping the application_2 from the bash terminal.

But when I’m trying to connect the nodes (using libcluster or simply using Node.connect(:application_1@application_1) ), nothing seems to work. I also try to set the same cookie for both application, Node.list is still empty.

Anyone have an idea on how I could simply connect those two nodes manually and even better using libcluster ?

Thanks :slight_smile:

So, let’s start with the manual connection in docker. Once it works on your side we can move to libcluster :slight_smile:

We run two iex in two different containers, connected to the same elixir network and passing the same ERLANG_COOKIE via env variable

First let’s create the bridge network (something that docker-compose creates automatically)

docker network create elixir

Then run the first container in a terminal, attaching it to the elixir network. The container’s name is app1 which is its domain name inside the network.

The command we run in the container is iex --sname app@app1 --cookie ${ERLANG_COOKIE}

$ docker container run -it  --rm --network elixir  \
  -e ERLANG_COOKIE="its_a_secret" \
  --name app1 \
   elixir:1.9 \
  bash -c 'iex --sname app@app1 --cookie ${ERLANG_COOKIE}'

then we run the second app2 container in another terminal

$ docker container run -it  --rm --network elixir  \
  -e ERLANG_COOKIE="its_a_secret" \
  --name app2 \
   elixir:1.9 \
  bash -c 'iex --sname app@app2 --cookie ${ERLANG_COOKIE}'

and we connect it to app@app1

iex(app@app2)1> Node.connect(:app@app1)
true
iex(app@app2)1> Node.list
[:app@app1]

So we can test connection with docker-compose using a simple script app.exs I just made for this case. We make available in the two containers via bind mounting

docker-compose.yaml

version: "3.7"
services:
  app1:
    image: "elixir:1.9"
    environment:
      ERLANG_COOKIE: "its_a_secret"
    volumes:
      - ./app.exs:/app.exs

    command: ["bash", "-c", "elixir --no-halt --sname app@app1 --cookie $$ERLANG_COOKIE app.exs"]
    # command: ["bash", "-c", "sleep infinity"]

  app2:
    image: "elixir:1.9"
    environment:
      ERLANG_COOKIE: "its_a_secret"
    volumes:
      - ./app.exs:/app.exs
    command: ["bash", "-c", "elixir --no-halt --sname app@app2 --cookie $$ERLANG_COOKIE app.exs"]
$ docker-compose up
app2_1  | I'm here! Sleeping for 2 seconds
app1_1  | I'm here! Sleeping for 2 seconds
app2_1  | [self is :app@app2]: :app@app1
app2_1  | connect (from :app@app2: true
app1_1  | [self is :app@app1]: :app@app2
app1_1  | connect (from :app@app1: true
app2_1  | nodes: [:app@app1]
app2_1  | ping :app@app1: :pong
app1_1  | nodes: [:app@app2]
app1_1  | ping :app@app2: :pong
app2_1  | ping :app@app1: :pong
...

@Zios does it work on your side?

4 Likes

First of all, thanks for the time you took in your response @alvises, that’s awesome :slight_smile: !

So, this weekend I tried to progress based on your answer. First of all, everything you said work for me. Based on that I tried to implement this stuff in my code.

The code I use is a little more complex because I’m using Distillery and Phoenix.

I changed rel/config.exs in order to use the the same cookie in my 2 applications :

environment :prod do
  set(include_erts: true)
  set(include_src: false)
  set(cookie: :"super_cookie")
  set(vm_args: "rel/vm.args")
  set(post_start_hooks: "rel/post_start_hooks")
end

And in vm.args:

-sname <%= release_name %>@application_1

I tried to use -name in the same way than Distillery’s documentation but it failed.

And it work ! So I will try to use Libcluster now :slight_smile:

1 Like

Today I had time to play with elixir releases, libcluster and docker-compose, trying to make them work together.

Have you tried the Cluster.Strategy.Gossip strategy? It works well in a docker bridge network and it finds the nodes automatically, without defining a fixed node list. This is great if you need to scale up or down without having to pass a fixed list of nodes every time.

Using the Elixir releases there is an RELEASE_NODE environment to set the sname option (is there maybe something similar with Distillery?), so my docker-compose.yaml file looks like this (webapp:3 is the image with the release of the sample app)

version: "3.7"

services:

  app1:
    image: "webapp:3"
    environment:
      RELEASE_NODE: app@app1

  app2:
    image: "webapp:3"
    environment:
      RELEASE_NODE: app@app2

and the lib/application.ex file is pretty simple

defmodule Webapp.Application do
  # See https://hexdocs.pm/elixir/Application.html
  # for more information on OTP Applications
  @moduledoc false

  use Application

  def start(_type, _args) do
    topologies = [
      default: [
        strategy: Cluster.Strategy.Gossip
      ]
    ]

    children = [
      {Cluster.Supervisor, [topologies, [name: Webapp.ClusterSupervisor]]},
      {Task, fn -> ping_nodes() end}
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: Webapp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  defp ping_nodes() do
    Process.sleep(1_000)
    Node.list()
    |> Enum.each(fn node ->
      IO.puts("[#{inspect(Node.self())} -> #{inspect(node)}] #{inspect(Node.ping(node))}")
    end)
    ping_nodes()
  end
end

Each node also start a Task to ping other nodes. And running docker-compose up I get this log

docker-compose up
Creating network "webapp_default" with the default driver
Creating webapp_app2_1 ... done
Creating webapp_app1_1 ... done
Attaching to webapp_app1_1, webapp_app2_1
app1_1  |
app1_1  | 23:32:25.284 [info]  [libcluster:default] connected to :app@app2
app1_1  | [:app@app1 -> :app@app2] :pong
app2_1  | [:app@app2 -> :app@app1] :pong
app1_1  | [:app@app1 -> :app@app2] :pong
app2_1  | [:app@app2 -> :app@app1] :pong
...

So, it works and it’s really easy to setup - but the problem with docker compose is that, as far as I know, is not possible to have just one app service and dynamically set a different RELEASE_NODE env variable for each replica. So we are forced to create a different service for each node…

I wrote an article few months ago about this, maybe it can be useful (especially if you are interested to deploy it with Kubernetes): Connecting Elixir Nodes with libcluster, locally and on Kubernetes. It shows how to connect phoenix chat nodes in kubernetes, but with mix, no releases. With Kubernetes is possible to dynamically set the env variables using the container’s IP, which is pretty cool :slight_smile:

2 Likes

Hi there,

I would like to achieve the same, but app1 and app2 would be started individually on two different VMs having their hostnames: foo and bar, for brevity. foo and bar are in the same vpc (network). How would the docker compose or the RELEASE_NODE be configured, so that I could use libcluster to create a cluster between app1 and app2?

Thank you!

Found the answer by myself, but replying here for brevity. The network_mode: host setting does the trick. After this, it is just setting properly the release and configuring libcluster with the appropriate strategy; I am using a custom one, in my case.

Thank you for the info offered by the original discussion, it helped!

1 Like