Distillery node defaults - should I be concerned about security?

I have a phoenix app deployed as distillery release.

By default, distillery sets node name to app_name@127.0.0.1 and cookie to app_name.

Do I need to be concerned about security here? Can anybody connect to the node and execute commands on my production machine?

From what I understand, in order to connect to such node, one should explicitly setup ssh tunnel to app_name@127.0.0.1 is this correct?

No, this is not correct. When starting the Erlang VM with a node name (short or long), a TCP listener is created on all network interfaces (unless you set inet_dist_use_interface; see below). You should assume that anyone who can reach that TCP port, from the local network or beyond, can try to connect. If they know, guess or brute-force the cookie, they gain complete control over the VM.

To harden your deployment:

  • If you are not using Erlang distribution at all, disable it altogether by not setting a node name
  • If you are only using Erlang distribution locally on the machine, e.g. to run Observer, set {inet_dist_use_interface, {127, 0, 0, 1}} in the Kernel app configuration; also set a strong cookie value, out of an abundance of caution
  • If you do plan to use Erlang distribution from other hosts (e.g. your dev machine), set it up for local use as described above and use an SSH tunnel with port forwarding
7 Likes

@voltone could you pls provide an example of how I can connect to such node bypassing ssh tunnel?

This is not about SSH tunnels. The issue is that the node name gives the false impression that the Erlang distribution protocol is only reachable over the loopback interface. In reality, the node is reachable on a randomly allocated TCP port through all other network interfaces as well.

Of course that does not mean the port is exposed to the Internet: I assume you only open up selected ports in your firewall. But a firewall only provides perimeter defense. The Erlang distribution protocol is too powerful to be protected only by a TCP port filter.

As for actually connecting to the node, e.g. from another machine in the local network: it is not trivial, but definitely possible. A normal iex -remsh will fail, because it will try to look up the node with the local EPMD on that machine (because of the ‘127.0.0.1’). But EPMD can be spoofed, and the part of the node name after “@” is only used to locate the node’s EPMD server, but is ignored during the node-to-node connection authentication.

EDIT: Correction, the domain part of the node name is used to discover the EPMD server and also in the handshake between the nodes; it is not used by EPMD, which only cares about the part before the “@”.

3 Likes

Also note, at least on older versions (unsure of now) you can run out of atom table entries via remote queries, don’t expose epmd at all except to what should be able to access it.

Actually, that issue was in the node itself, not EPMD (which is written in C, and therefore does not deal with atoms).

And no, the issue was not resolved. In OTP 21 the node whitelist (:net_kernel.allow/1; docs) will now be checked prior to creating an atom for the remote node name. But no whitelist is active by default, and use of a whitelist may not be practical in many cases.

Earlier, in OTP 20, a check was added to look for a “@” character inside the node name prior to conversion to an atom. Due to this, the code examples in my original blog post on the topic no longer work as-is, but it’s trivial to work around that, of course.

So that’s a DoS risk, which is another reason to disable the distribution protocol, or at least bind it to the loopback interface only.

2 Likes

@voltone Does this mean that all Distillery apps that have not been given special configuration are currently susceptible to a clever attacker that sniffs ports?

If so, could you give us a clear explanation of what needs to be altered in our configurations?

@Qqwy when you build release, it shows you a warning if you don’t set a cookie.

With distillery, you can’t disable distributed erlang, because it uses it to start/restart node & other commands. So the only option you have is to setup firewall and use inet_dist_use_interface

In my case, I’m using docker container and there’s no need to adjust firewall, since the only port I expose is 80.

I’ve also added this to my config.exs and increased cookie length as @voltone blog post suggests.

config :kernel, inet_dist_use_interface: {127, 0, 0, 1}
2 Likes