Secure way to remote access to a Nerves device using nodes distribution with epmd?

[ivanhercaz]

Hi everyone!
I am really interested in security issues and remote access to Nerves devices, because I am working in a project in which we are going to have many devices in different locations and I would like if it would be possible to provide support in a safe way.

At this moment the system I am developing consists in two parts:

• An Elixir application in a server that runs a iex shell with a node, a security cookie and TLS enabled.
• A support system in the Nerves devices, part of the application, in which I supervised the starting of a node (start, set the cookie and check TLS is working) and its connection with the server’s node.

So I can access to this server, check which nodes are connected and apply RPC calls and even open a remote iex shell. It is a very comfortable system, because I can manage the devices, give support, update firmware, etc., without many complications.

However, as far as I know from what I read, epmd still manages the communications between the nodes in an unencrypted way.

In the blog post Erlang Elixir Node Security Flaws, @Andrei suggest some solutions:

1. Setting up Erlang to use TLS/SSL.
2. Using SSH.

I already configured the first point in both parts of the system, the application in the server and the Nerves device. But I am still thinking about how to setup a reverse SSH tunnel from the Nerves device to the remote server.

I have tried a lot and the result is a lot of dirty code hehe! But I am still looking for a good way to do it.

I am trying to use :ssh module and the functions :ssh.connect/3 and :ssh.tcpip_tunnel_to_server/5 to setup the tunnel. Just like a test (not as part of the Elixir application in the Nerves device), I tried to run in the Nerves iex shell these commands:

iex(device_node@nerves-hostname)7> {:ok, ssh_conn} = :ssh.connect(ip_address, 22, [user: user, password: pass])
{:ok, #PID<0.1826.0>}

iex(device_node@nerves-hostname)8> {:ok, tunnel} = :ssh.tcpip_tunnel_to_server(ssh_conn, 0, "localhost", 4369, "localhost", 4369)
** (MatchError) no match of right hand side value: {:error, :bad_connect_to_address}
    (stdlib 4.2) erl_eval.erl:496: :erl_eval.expr/6
    iex:8: (file)

I think that the main possibility is that I don’t know how to setup and work with SSH tunnels with Elixir and Erlang. Even just with ssh command I have many to learn yet. Thus I want to ask you for help about this topic: what would be the best way to set up an SSH tunnel from the Nerves device to the remote server?

In addition, what do you think about this approach to provide remote support to Nerves devices from a server?

I am really interested in this topic and all the related aspects, but especially in the matter of security.

Regards,
Iván

[jjcarstens]

On first glance, the arguments for the tunnel are wrong according to the docs

So first step is to retry that with something like this:

:ssh.tcpip_tunnel_to_server(ssh_conn, 'localhost', 4369, 'localhost', 4369)

Also note that you need Erlang strings, which are charlists in Elixir

Outside of that, I’ll have to think more on it. This is not a situation I’ve needed to deal with. Is your primary goal to access nodes from totally remote networks? (As in 2 geographically distant locations) Or is the server and Nerves device within the same local network?

If it’s the first options, a lot of times many reach for other options for secure communication using TLS in message delivery, like MQTT, rather than deal with the nuances of Erlang distribution outside of a local network

[Andrei]

Hi

I haven’t set up multi-nodes on Nerves, mostly, each raspberrypi I use is separate, communication is done via HTTP, API calls or something else. See if you REALLY need to have those connected, otherwise use a different solution.

The SSH security could be in theory achieved via erlang’s SSH but I’m not sure if all the features you need are supported. I remember when I experimented, something was missing. (I played around with SSH for a plethora of reasons)
What I meant about SSH in my article was that you create a reverse proxy OR a listener port with SSH commands.

Mostly, it can be achieved having a listener port locally for
ssh -fNT -L 40000:localhost:5432 user@yourdomain.net -p 2327

-L opens the port 40000 on localhost for the reomte port 5432 (This is How I access a secured postgresql instance via a sever without exposing the port on the remote machine)
There is also the -R option which could be more what you want. Again, it would need to be run on each instance to

1. make epmd port available
2. You need to make epmd give you a predefined port so you can make that one available as well for connection via ssh. I remember playing around with this some time ago

This means you’ll need to run the commands on each nerves device and fiddle around with the ports.
I know that nerves is booting directly in erlang so you either explore the ssh module or try installing linux or compiling ssh separately into the firmware if the erlang SSH is unresponsive.
This would require a smart script to fiddle around as with each node added you’d theoretically need to keep track of connections & tunnels.

I remember trying to work on the above a few years back but can’t find my complete notes
This might also be helpful as an alternative

Erlang Distribution Without epmd

There’s also the epmdless approach to static ports GitHub - tsloughter/epmdless: Distributed Erlang without the EPMD

Good luck. Let us know what worked:)

[ivanhercaz]

@jjcarstens

On first glance, the arguments for the tunnel are wrong according to the docs

So first step is to retry that with something like this:

:ssh.tcpip_tunnel_to_server(ssh_conn, 'localhost', 4369, 'localhost', 4369)

Also note that you need Erlang strings, which are charlists in Elixir

Oh! That’s true. I forgot to use charlist. And… it works! Now if I do:

:ssh.tcpip_tunnel_to_server(ssh_conn, 'localhost', 4369, 'localhost', 4369)

I get a {:error, :eaddrinuse}, but that’s normal because the local port 4369 is already in use with the epmd I have in Nerves. Now I am going to play with that! Of course, if I use an available port I get {:ok, 1111}.

@jjcarstens

Outside of that, I’ll have to think more on it. This is not a situation I’ve needed to deal with. Is your primary goal to access nodes from totally remote networks? (As in 2 geographically distant locations) Or is the server and Nerves device within the same local network?

As it is a work in progress project and we haven’t set up anything in production yet, I am thinking about what possibilities I can offer. If I have the possibility to set a secure communication between the nodes and a remote server (outside the local network of Nerves), I will continue to develop this idea, because the nodes distribution provide a really good useful tools, like RPC calls and the Erlang’s JCL (Job Control Mode) to open remote shells.

@jjcarstens

If it’s the first options, a lot of times many reach for other options for secure communication using TLS in message delivery, like MQTT, rather than deal with the nuances of Erlang distribution outside of a local network

That’s another possibility, of course. Set a secure connection with another messaging protocol, although it wouldn’t have the advantages I mentioned above about the distributed nodes. But I still worried about the security, so if it isn’t possible with distributed nodes + TLS + an SSH tunnel, I am going to reconsider its usage and probably propose a similar alternative.

[ivanhercaz]

@Andrei

I haven’t set up multi-nodes on Nerves, mostly, each raspberrypi I use is separate, communication is done via HTTP, API calls or something else. See if you REALLY need to have those connected, otherwise use a different solution.

The SSH security could be in theory achieved via erlang’s SSH but I’m not sure if all the features you need are supported. I remember when I experimented, something was missing. (I played around with SSH for a plethora of reasons)
What I meant about SSH in my article was that you create a reverse proxy OR a listener port with SSH commands.

Yes, I have to test now that I learnt I was having two errors in my code as @jjcarstens mentioned in his message, if I don’t get to achieve this, as I told in my reply to him, I am going to use another alternative, of course, because I need secure communications.

What provides the nodes distribution is the possibility to run RPC calls and open remote shells to the devices in a really easy way, but if that compromises the security… I prefer to reconsider the use of distributed nodes.

@Andrei

Mostly, it can be achieved having a listener port locally for
ssh -fNT -L 40000:localhost:5432 user@yourdomain.net -p 2327

-L opens the port 40000 on localhost for the reomte port 5432 (This is How I access a secured postgresql instance via a sever without exposing the port on the remote machine)
There is also the -R option which could be more what you want. Again, it would need to be run on each instance to

I am going to try to replicate it with the :ssh Erlang’s module, specifically connection the device and using the function I already know how works hehe.

@Andrei

There’s also the epmdless approach to static ports GitHub - tsloughter/epmdless: Distributed Erlang without the EPMD

Good luck. Let us know what worked:)

Thank you very much for sharing the article of Erlang Solutions and the epmdless, I will take a look at both.

If I succeed, I will share here what solution I apply to this issue.

Thank you very much!

Edit 1
@jjcarstens, taking advantage of the fact that you are participating in the thread and that NervesHub 2.0 has released, I am thinking about how NervesHub deal with the connection to the devices. If I understand correctly how NervesHub works, it is not in the same network of the devices, it is on internet.

I read a bit about that, because I am interested in NervesHub too, and it seems the devices communicate with NervesHub using Phoenix Channels. How you NervesHub secure the communication?

Excuse me if what I am asking is already available to read and inspect in some place, but recently I am usually reading a lot of resources and I miss some readings.

And sorry for the double posting! I read a recommendation of Elixir Forum about not reply in multiple posts when I was going to ask you that. I tried to solve it pasting the last post with the previous one, but then I can’t delete the last post, so finally I leave it as it is, and I will take it into account the next time!

Edit 2
Wait a moment! I have deployed a local instance of NervesHub 2.0 to check how it looks and I really like to deploy a production one hehe but it seems really interesting.

[jjcarstens]

NervesHub uses client side SSL (so certificate validation both ways, on client and server). The peer has to be trusted on both ends to make the connection

The remote console sends packets through the websocket channel using TLS, so is typical encrypted data back and forth.