Alright, that was quite the ride but I think I finally figured it out.

TLDR

adding this to rel/env.sh.eex fixed it for me

#!/bin/bash

export RELEASE_DISTRIBUTION=name

Ok, but why?

Note that I am still new to the Elixir/BEAM ecosystem so I might be wrong on some things. If something doesn’t make sense, please correct me.

The bin/<app-name> remote command connects to your running Phoenix app using a remote shell. As far as I understand, this starts a new BEAM node and connects to your existing Phoenix node using the magic of BEAM. This is the same thing that you would do to connect two BEAM nodes together in a distributed cluster. The only difference is that this is all happening on the same host.

The way the BEAM discovers how to connect to nodes is with epmd (Erlang Port Mapper Daemon). When a BEAM node starts it registers itself with epmd using it’s name. By default, this is the name of your Phoenix app when using Phoenix releases. If you then want to connect remotely to the node, you can use one of two formats to discover it, name or sname.

sname is used by default. When using sname, you can just pass the name of the app to epmd and it will try to look for that node on you local machine. The format for node names is name@host but when using sname, the @host portion is implicit since it is always the localhost.

name on the other hand is used when you also want to connect to a BEAM node on local and remote machine. It takes the format of name@host where @host is the domain name or ip address of the other machine that holds the node you want to connect to.

Now what I believe happens is that when you use sname, it seems to default to using the top level domain of whatever hostname is and uses that as the implicit @host part. In most cases, this is fine. But if the hostname is set to an ip address, epmd will take what it thinks is the top level domain and take the first 1-3 digits of the ip address. So for example, if you have a hostname of 123.456.789, epmd will take 123 as the “top level domain” and use that as the implicit @host. (ex: app@123)

The problem this causes is that when epmd tries to resolve the 123 host, it leads to no where. Resolving number-only domains results in weird behaviours most of the time. And so epmd just can’t find the nodes on localhost since it tries to look for node on a host that doesn’t exist.

Now, Kamal sets the --hostname flag on the docker container to the IP of the server/role that you specified in deploy.yml followed by what seems to be a random hash, ex: 123.456.789-hgjkagh. This causes the problem explained above and makes epmd unable to find your localhost nodes.

So now here is the solution, when RELEASE_DISTRIBUTION is set to name, the default @host (In Phoenix release at least) will be the full hostname and not just the top level domain part. This works because when you pass the --hostname flag to Docker, it will add an entry to /etc/hosts pointing that hostname to the IP of the docker container on the network. This enables epmd to resolve that hostname to you current docker container, and is then able to find the node to connect to.

Hope this helps, if you know more about this topic and believe that I made a wrong assumption, please let me know, I would be very interested to learn.

I still didn’t find the fix but the issue seems to be occuring because Kamal sets the hostname flag on the docker container to you server IP followed by a hash.

I believe this causes the started iex shell to be unable to connect to the server since they are on two different network or something like this.

I will try to debug some more tomorrow and see if I can find a solution

Not really. This works when the --hostname flag doesn’t start with numbers followed by a .. This works when you don’t set a hostname, or when the hostname contains numbers and letters.

This would work: --hostname sslmoon;
This would also work --hostname 123sslmoon;
But this would’t work: --hostname 123.sslmoon;
And this also would’t work: --hostname 123.456.789-sslmoon (which is essentially what Kamal gives us).

You can try it by starting the docker container locally and specifying the hostname