Device won't handshake with nerves-hub using nerves_hub_link

Hi,

I’m unable to let my device handshake with nerves-hub using nerves_hub_link. The dmesg() of RingLogger don’t complain about it, so it is hard to debug for me.

The Supervisor is up:

iex(...)> Supervisor.which_children(NervesHubLink.Supervisor)
[
  {NervesHubLink.UpdateManager, #PID<0.1766.0>, :worker,
   [NervesHubLink.UpdateManager]},
  {NervesHubLink.DeviceChannel, #PID<0.1765.0>, :worker,
   [NervesHubLink.DeviceChannel]},
  {PhoenixClient.Socket, #PID<0.1764.0>, :worker, [PhoenixClient.Socket]},
  {NervesHubLink.Connection, #PID<0.1763.0>, :worker,
   [NervesHubLink.Connection]}
]

The connection is disconnected (taking the pid from above):

iex(...)> :sys.get_state(pid(0, 1763, 0))
{:disconnected, 39}

Eventually the NervesHub Connection does state it is disconnected too long:

iex(...)> NervesHubLink.Connection.check
{:error, {:disconnected_too_long, 39}}

The problem is, I don’t have a clue where it goes wrong. I followed the instructions on https://docs.nerves-hub.org/

I don’t have a NervesKey, so I generated a self-signed root-ca and signing-ca, which signed the certificate for the devise which I use in the config.ex:

config :nerves_hub_link,
  socket: [
    json_library: Jason,
    heartbeat_interval: 45_000
  ],
  ssl: [
    certfile: "config/certs/self-signed-device.crt",
    keyfile:  "config/certs/self-signed-device.key"
  ],
  fwup_public_keys: [:devkey]

I also uploaded the root-ca and signing-ca to nerves-hub as Certificate Authorities.
This is all done in step https://docs.nerves-hub.org/nerves-hub/setup/adding-nerveshub-to-your-project

Later on the devise is created using mix nerves_hub.device create which also generates certificates. So now I have 2 device certificates: one self-signed using openssl with my own self-signed CA, and one generated by the mix nerves_hub.device create command. I don’t understand why I have two completely, so maybe the problem lies there.

So I guess, the main question is: how to approach this debugging?

Thanks,
Bas

TL;DR - SSL is hard to debug and cryptic (get it? :wink:). I typically look for another pair of eyes when first running into SSL issues.

Looking at your setup, I can see 2 things to check:

  1. When you use your own CA signer cert, you also need to include that in the connection request (in cacerts key) along with the device cert and device key generated with it. Just remember to add it to the list of CA’s, not replace the list - Because there are also some server CA intermediates in there used for the basic connection to the server. Look at my poser setup as an example for whats needed with custom Signer CAs
  2. If that is your exact config, then its also broken on the device. Because the config is evaluated at compile time on host, but those values are fetched at runtime on device. So the "config/certs/*.{cert|key} relatives aren’t going to point to the correct location on device. That said, you might be accounting for that in your code, but I at least wanted to point out in case thats an issue as well
1 Like

Hi @jjcarstens, thanks for the help!

Do you know why there are no SSL rejection errors from nerves-hub-link in the device’s RingLogger? Logging would help a lot, pointing into the general directions about what goes wrong.

I’ll try that! But I don’t really understand why the CA has to be sent by the device as well. The device certificate already contains the CA signature and the CA is uploaded/registered on Nerves Hub. But I also see now that there is a default “NervesHub Device CA” as well, I didn’t know about that before. So I’ll remove my own CA stuff and try to stick to the defaults (I have generated a root-ca and signer-ca, but I don’t really understand the difference… I really need to read up on SSL stuff). I’ll definitely take a look at poser as well!

Ah, totally missed that indeed. I was already wondering which files were included in the build and which are not. I also ordered an ATECC608A breakout, which would bring my setup closer to existing docs as well. I didn’t get the NervesKey one though. I’ll do that later (but shipping to Europe is a drag). But I assume the I2C interface is identical.

I’ll keep trying different approaches, and update this thread with results/failures.

… removing intermediate post, to keep the thread tidy. Can only delete one post per 24 hours, so editing to this as alternative.

Hi @jjcarstens,

I’ve tried another approach using nerves_key (the project, not the hardware), rebuilding the CA and device certs. My device serial number is 9794.

In a separate nerves_key project:

mix nerves_key.signer create nerveskey_prod_signer1
mix nerves_hub.ca_certificate register nerveskey_prod_signer1.cert
mix nerves_key.device create 9794 --signer-cert nerveskey_prod_signer1.cert --signer-key nerveskey_prod_signer1.key

Copy all the keys to “priv” of the home_sensor project.
Based on poser/lib/poser/configurator.ex:

  def build(config) do
    priv_dir = Application.app_dir(:home_sensor, "priv")
    certfile = Path.join(priv_dir, "9794.cert")
    keyfile = Path.join(priv_dir, "9794.key")

    signer =
      Path.join(priv_dir, "nerveskey_prod_signer1.cert")
      |> File.read!()

    update_config(config, certfile, keyfile, signer)
  end

But I get the error on Application.get_env(:nerves_hub_link, :ca_certs) which return nil. So I worked around that with:

  defp cacerts() do
    case Application.get_env(:nerves_hub_link, :ca_certs) do
      [] = list ->
        list
        |> Path.join("*")
        |> Path.wildcard()
        |> Enum.map(&to_der/1)
      nil ->
        []
    end
  end

building and burning the firmware:

MIX_TARGET=rpi0 mix firmware
MIX_TARGET=rpi0 mix nerves_hub.device burn 9794

But still no connection to NervesHub.

I think maybe the problem lies in the MIX_ENV. I’m only setting the MIX_TARGET=rpi0, so the environment defaults to dev.

Nerves environment
  MIX_TARGET:   rpi0
  MIX_ENV:      dev

NervesHub server: api.nerves-hub.org:443
NervesHub organization: basvanwesting
Burning firmware

The fact that the NervesHubLink.Connection is disconnected without complaining, maybe suggests it is not even trying to connect.

Some progress! I’m getting TLS rejection errors in the device RingLogger:

TLS :client: In state :wait_cert at ssl_handshake.erl:1950 generated CLIENT ALERT: Fatal - Unknown CA

So there is something wrong with the CA. But I’ve registered the signer CA properly. Maybe the Application.get_env(:nerves_hub_link, :ca_certs) returning nil means some CA’s are missing. But I didn’t change anything from the defaults concerning that.

One thing to note is that the command mix nerves_hub.device cert list 9794 results in an empty list:

NervesHub server: api.nerves-hub.org:443
NervesHub organization: basvanwesting
Local NervesHub user password:

Device: 9794
Certificates:
------------

How do I register the device cert/key?
I tried using the explicitly with during burn:

MIX_TARGET=rpi0 mix nerves_hub.device burn 9794 --cert priv/9794.cert --key priv/9794.key

But same results.

SSL is passed down to Erlang and handled there as part of the connection attempts. If you want more logging, pass log_level: :debug to the SSL options. Either in your configurator or config like:

config :nerves_hub_link, ssl: [log_level: :debug]

This is also one of the crux that makes debugging SSL so hard. NervesHubLink just has no way to know why SSL is failing. It just fails, and minimal logs are put out (or I don’t know how to get more info here)

Application.get_env(:nerves_hub_link, :ca_certs) is definitely the issue here. It being empty will cause the server to reject any requests. You need all the NervesHub intermediate CA’s plus your signer CA. Also, using this Application config is sort of my convention because I do a lot of SSL cert swapping/testing locally, but its sort of a poor example. For you, I would recommend removing an ca_certs values from your configs. Then change your configurator line to reference already included CA certs:

- [signer_der | cacerts()]
+ [signer_der | NervesHubLink.Certificate.ca_certs()]

I’ve also updated the docs for this

You don’t need to do this. On first successful connect, the device cert will be added to the device in NervesHub. This is part of the pattern of client-side SSL.

NervesHub uses both Server side SSL and Client side SSL. In a typical server setup, the client requests data from a Server and the client decides if it trusts the server by referencing the CAs/intermediates given from the server with what the client trusts.

For client side, flip that around. You device is attempting the connection and is saying the Server must verify the device connection as well. So in this flip, the server needs to decide if it trusts this incoming device by checking its CA against what it allows (which is why you must register the device signer CA). But you also still need all the intermediates from NervesHub for the typical server->client connection checks as well.

Here’s a little visual breakdown of what is needed in the cacerts of the request:

[
  device_signer_ca, # For Client -> Server (client-side SSL)
  NervesHub-root-ca, # For Server -> Client  (server-side SSL)
  NervesHub-server-ca, # For Server -> Client  (server-side SSL)
  NervesHub-device-ca, # For Server -> Client  (server-side SSL)
]

Hi @jjcarstens

Success! The missing step was the NervesHubLink.Certificate.ca_certs() part.
Thanks very much for all the help!

Regards, Bas