Elixir release `myapp remote` returns `:nodedown`

I’ve built a release of my app using mix release. The release is built into a docker container, and deployed, and the app runs fine.

When I docker exec -it xxxxxxx bash to get inside the container and run bin/myapp remote I get:

Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]

Could not contact remote node staging@10.0.12.121, reason: :nodedown. Aborting...

epmd is running, and shows my app:

# ./erts-13.2.2/bin/epmd -names
epmd: up and running on port 4369 with data:
name staging at port 9001

Additionally, I’ve got some code in env.sh.eex to set the the node name to the external IP address so clusters can be formed with other machines:

token=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" || true)
if [ -n "$token" ]; then
  hostname=$(curl -H "X-aws-ec2-metadata-token: $token" -v http://169.254.169.254/latest/meta-data/local-ipv4)
else
  hostname="127.0.0.1"
fi
export RELEASE_DISTRIBUTION=name
export RELEASE_NODE="$ERLANG_CLUSTER@$hostname"

which results in the node name of staging@10.0.12.121.

netstat shows the apps listening on 0.0.0.0:

# netstat -ln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:8083            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8082            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:9001            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN
tcp        0      0 :::4369                 :::*                    LISTEN
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node Path

I can connect to port 9001 with netcat:

# nc -v localhost 9001
localhost (127.0.0.1:9001) open

Why doesn’t remote work?

If I remove the code setting the node name, I can run remote:

I think the problem for this container was that I didn’t map the epmd/distribution ports onto the host, so it was trying to connect to the host’s IP which was closed.

I have a complicated situation where we have an umbrella app where I generate 6 releases from it, some of those releases use Erlang distribution, and the others don’t. I was trying to setup everything to work in both situations, but Erlang distribution being designed before the cloud makes it difficult.

In the end, I found it easier to find another way to what I was doing that required Erlang distribution, and remove it.

1 Like