What is the proper certificates configuration for NervesHub 2.0 and NervesHubLink?

Recently it was merged a really wonderful PR in the current main branch of NervesHub, that eases the self hosting setup and, I can confirm after test it, that it really eases the deployment. It is friendly and easy to manage.

It is part of the development of the NervesHub 2.0: Introducing NervesHub 2.0. Thus it is important to keep in mind that the documentation is also in progress of update, although the NervesHub team is doing a really great work with both parts, the development and the documentation about it and the deployment process.

So, thanks to this PR and the documentation about the self hosting setup provided in the readme of the repository, I got it deployed on my cloud server!

Now I am in the step of trying to link my Nerves devices to my NervesHub instance and, although I understand the theory and practically everything about the related and necessary stuff to use NervesHubLink, I feel bit lost with the certificates configuration. At this moment I am getting this error on the server side:

May 11 03:01:05 nerveshub nerves_hub[529627]: 03:01:05.118 level=notice TLS :server: In state :wait_cert received CLIENT ALERT: Fatal - Unknown CA

And this error with any mix task of NervesHub (for example, mix nerves_hub.ca_certificate list):

$ mix nerves_hub.ca_certificate list
NervesHub server: nerveshub.DOMAIN.EXT:4001
NervesHub organization: ORG

13:45:47.397 [notice] TLS :client: In state :wait_cert at ssl_handshake.erl:2111 gener
ated CLIENT ALERT: Fatal - Unknown CA

Failed to list CA certificates 
reason: {:error, {:tls_alert, {:unknown_ca, 'TLS client: In state wait_cert at ssl_han
dshake.erl:2111 generated CLIENT ALERT: Fatal - Unknown CA\n'}}}

In my environment variables I have NERVES_HUB_HOST pointing to my instance, NERVES_HUB_PORT as 4001, the one for the devices endpoint and then I have NERVES_HUB_KEY and NERVES_HUB_CERT with the key and cert generated in the NervesHub setup I did in the server. Of course, I also have a generated token for my user in NERVES_HUB_TOKEN.

But I think the problem is that I am mixing concepts related with the certificates and I am configuring it in the wrong way. Letā€™s see what I did in the NervesHub setup.

Before the deployment step I had to generate the certificates for my NervesHub instance, so I follow the next instructions, but of course adapted to my domain:

$ openssl genrsa -out ca.key 2048
$ openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.pem -subj '/OU=NervesHub/'
$ openssl genrsa -out device.nerves-hub.org-key.pem
$ openssl req -new -key device.nerves-hub.org-key.pem -out device.nerves-hub.org.csr -subj '/CN=device.nerves-hub.org/'
$ nvim device.nerves-hub.org.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = device.nerves-hub.org
$ openssl x509 -req -in device.nerves-hub.org.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out device.nerves-hub.org.pem -days 825 -sha256 -extfile device.nerves-hub.org.ext
$ sudo mv device.nerves-hub.org-key.pem /etc/ssl/
$ sudo mv device.nerves-hub.org.pem /etc/ssl/

So I replaced all device.nerves-hub.org by device.nerveshub.DOMAIN.EXT (DOMAIN.EXT is correct, I am just hiding it because I prefer to not have links to it). And in my nerves_hub.env I have the next related with the host/ports for each endpoint, that I think is correct after read several times the instructions:

HOST=nerveshub.DOMAIN.EXT
PORT=4000
URL_SCHEME=https
URL_PORT=443

DEVICE_HOST=device.nerveshub.DOMAIN.EXT
DEVICE_PORT=4001

API_HOST=nerveshub.DOMAIN.EXT
API_PORT=4002
API_URL_PORT=443

And finally, in my Nerves device I added NervesHubLink package poiting to its git repository and specifying the branch nerves-hub-2.0 (I also tested it with main and the latest version available in Hex). In config.exs I set the next:

config :nerves_hub_link,
  device_api_host: "nerveshub.DOMAIN.EXT",
  device_api_port: 4001,
  device_api_sni: "nerveshub.DOMAIN.EXT",
  ssl: [
    cert: "priv/ca.pem",
    keyfile: "priv/ca.key",
  ]

cert: "priv/ca.pem" corresponds with NERVES_HUB_CERT and key: priv/ca.key corresponds with NERVES_HUB_KEY, generated with the previous openssl commands in the server:

$ openssl genrsa -out ca.key 2048
$ openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.pem -subj '/OU=NervesHub/'

And I know that I should provide cacerts option too, but I am really lost with the certificates hehe.

My theory is:

  • I am not understanding what cert, key and ca I must use in my Nerves device.
  • I may be confused in the certificates generation step of NervesHub and I may be misunderstood something important for the step of linking a device with the hub.

This has become a long thread and probably quite confusing and with different problems, sorry about that hehe. I just hope someone can enlighten me in the darkness that I have been involved in with the certificates.

Edit: I am reading the next threads looking for a possible answer to my error/confusion:

Thanks @ivanhercaz !

I think there are a few steps to take:

  • Try curl https://nerveshub.DOMAIN.EXT/api/health just as a simple connectivity check
  • The NERVES_HUB_CERT (ca.pem) is your server certificate that it is present to the device on connect (like any other HTTPS server/website you visit). On device, that needs to go in :cacerts option as a list of DER format certs (or if you have the file, you can just use cacertfile: "ca.pem")
  • You need to create a signer CA certificate used for creating device certificates. Admittedly, the documentation around this is dated and sparse. The easiest steps would be to use the task in GitHub - nerves-hub/nerves_key: NervesKey info and support for Elixir
    • mix nerves_key.signer create some_signer_name
  • Then, create a device certificate
    • mix nerves_hub.device cert create SERIAL --signer-cert /path/to/some_signer_name.cert --signer-key /path/to/some_signer_name.key
    • This device SERIAL needs to be created in NH with the newly generated certificate. Its probably easiest to do this in the UI and manually upload the certificate to it
  • Register your some_signer_cert signer CA with NH either in the web UI or with mix nerves_hub.ca_certificates register /path/to/some_signer_cert.crt
  • mix nerves_hub.ca_certificates will first need mix nerves_hub.user auth
    • Ensure your NervesHubUserAPI config has the ca.pem for the server in the options as well
# cacerts needs to be DER formatted. Probably easiest to use
# `cacerts =Enum.map(X509.from_pem("ca.pem"), &X509.Certificate.to_der/1)` and put that
# result in the cacerts key below
config :nerves_hub_user_api, ssl: [cacerts: cacerts]
1 Like

Thank you for your detailed answer, @jjcarstens!

{"status":"UP"}, what makes me happy because I donā€™t think in test it hehe.

Okay, I understand it. So I am going to copy the ca.pem I generated in the server and test the option of use it in :cacertfile in :ssl configuration of NervesHubLink, without converting it into DER format.

All right, I have installed nerves_key package in my project and generate the signer cert and the private key.

I uploaded the signer.cert for the device into the certificates section of NervesHub (web) and then I use the certificates serial to run:

mix nerves_hub.device cert create serial --signer-cert signer.cert --signer-key signer.key

Generated APP app
NervesHub server: nerveshub.DOMAIN.EXT:4001
NervesHub organization: asi
Creating certificate for 49:B6:9C:66:9B:A6:0E:DE:32:6F:9F:7A:9D:C0:0E:6F
Signer cert path: signer.cert
Signer key path: signer.key
Finished

It seems that it works!

But in these steps I am having an error, a different one! What it is very helpful because something has changed following your instructions:

16:01:15.942 [notice] TLS :client: In state :wait_cert at ssl_handshake.erl:2113 gener
ated CLIENT ALERT: Fatal - Handshake Failure
 - {:bad_cert, :hostname_check_failed}
Failed to list CA certificates 
reason: {:error, {:tls_alert, {:handshake_failure, 'TLS client: In state wait_cert at 
ssl_handshake.erl:2113 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,h
ostname_check_failed}'}}}

And in the server side:

May 11 15:01:15 nerveshub nerves_hub[529627]: 
15:01:15.988  level=notice TLS :server: In state :wait_cert received CLIENT ALERT: Fatal - Handshake Failure

When I tried your proposal for the cacert in the configuration of NervesHubUserAPI I get:

** (UndefinedFunctionError) function X509.from_pem/1 is undefined (module X509 is not 
available)

So I tried a workaround and convert ca.pem into ca.der using openssl x509 -in config/ca.pem -out config/ca.der -outform DER and then I leave the configuration as follows:

ca_cert = "config/ca.der" |> File.read!()

config :nerves_hub_user_api, ssl: [cacerts: [ca_cert]]

config :nerves_hub_link,
  device_api_host: "device.nerveshub.DOMAIN.EXT",
  device_api_port: 4001,
  device_api_sni: "device.nerveshub.DOMAIN.EXT",
  ssl: [
    cert: "config/ca.pem",
    keyfile: "config/ca.key",
    cacertfile: "config/ca.pem"
  ]

I also tried to change device_api_host and device_api_sni removing the device. subdomain part, but that doesnā€™t works, although I understand the mix task get the env variable NERVES_HUB_HOST for this.

At least this error is more descriptive. It seems to me that I have read something about this error in the threads that I mentioned in my first post, I will check them.

Your device SSL settings are still wrong. cert and key need to be the device certificate and key you generated with mix nerves_hub.device cert create. Also cert and key expect the DER formats. If you want to use files, use the certfile and keyfile arg

config :nerves_hub_link,
  ssl: [
    certfile: "path/to/device-cert.pem",
    keyfile: "path/to/device-key.pem"
  ]

Oh! All right, and now I discovered that I was a bit lost (more indeed haha!), because the device certificate generated is available at ~/.nerves-hub. When I read your message I was looking for them in the same directory where I ran the mix task and I donā€™t remember the ~/.nerves-hub directory.

And for the last example you gave me, I understand the cacertfile should not be included. Although with or without it, and the changes you suggested me, it doesnā€™t work yet and returns the same error:

config :nerves_hub_link,
  device_api_host: "device.nerveshub.DOMAIN.EXT",
  device_api_port: 4001,
  device_api_sni: "device.nerveshub.DOMAIN.EXT",
  ssl: [
    certfile:
      "/home/ivanhercaz/.nerves-hub/SERIAL-cert.pem",
    keyfile:
      "/home/ivanhercaz/.nerves-hub/SERIAL-key.pem"
  ]

Edit: I also tried converting the key and the cert to der previously and using :key and :cert, but nothing changes.

You do still need the cacertfile. Iā€™m assuming since these are paths to files that you are trying to run NervesHubLink on host? Or are you making firmware and putting this on device?

The error persists.

I am running this on my local machine but not with MIX_TARGET=host, I specify my target, because if not I get ** (Mix) The task "nerves_hub.ca_certificate" could not be found, what I donā€™t understand because nerves_hub_link is not marked with the targets key, so the tasks of NervesHubCLI derived from NervesHubLink should be available in whatever target I set, no?

Edit

I think I may still not understanding what cacerts should have, at least reading this:

On Device won't handshake with nerves-hub using nerves_hub_link - #8 by jjcarstens.

Edit
I think my problem is with the castorefile, because if I am not wrong, this CA must have the root ca certificate and the certificates for nerveshub.DOMAIN.EXT, but in my ca.pem there is only one certificate.

I need to review it with patience and I may be need to restart the generation of the certificates in the server, because it is possible that I type something wrong.

One doubt I have with the openssl commands I posted in the first post:

$ openssl genrsa -out ca.key 2048
$ openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.pem -subj '/OU=NervesHub/'
$ openssl genrsa -out device.nerves-hub.org-key.pem
$ openssl req -new -key device.nerves-hub.org-key.pem -out device.nerves-hub.org.csr -subj '/CN=device.nerves-hub.org/'
$ nvim device.nerves-hub.org.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = device.nerves-hub.org
$ openssl x509 -req -in device.nerves-hub.org.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out device.nerves-hub.org.pem -days 825 -sha256 -extfile device.nerves-hub.org.ext
$ sudo mv device.nerves-hub.org-key.pem /etc/ssl/
$ sudo mv device.nerves-hub.org.pem /etc/ssl/

In the commands we use device.nerves-hub.org, what I replaced by device.nerveshub.DOMAIN.EXT, and the ca.pem I am using in my configuration is the one that is returned by the second openssl command:

$ openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.pem -subj '/OU=NervesHub/'

So I do not rule out the possibility that I am confusing the file.

I have tried to compose cacerts as a list of DER files with the device_signer, the ca.pem generated in the NervesHub setup and with nerveshub.DOMAIN.EXT.pem, but it doesnā€™t work and I am really lost about how to compose the cacerts needed.

Edit
I tried to replicate what @jjcarstensā€™ Poser does, but without luckā€¦

Edit
I am going to clean everything and restart the process hehe, I think I have many tangled things at this moment. But I am going to write step by step as a breadcrumb path.

So the walkthrough I followed is the next one:

  1. Format the server to start all the process from zero.
  2. ./bin/remote setup to setup the server.
  3. Formateo el servidor, empiezo con Ć©l desde cero.
  4. Create the database, user and assign password.
  5. Generate the ca.key with openssl genrsa -out ca.key 2048.
  6. Generate the ca.pem with openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.pem -subj '/OU=NervesHub/'.
  7. Generate nerveshub.DOMAIN.EXT-key.pem with openssl genrsa -out nerveshub.DOMAIN.EXT-key.pem
  8. Generate nerveshub.DOMAIN.EXT.csr with openssl req -new -key nerveshub.DOMAIN.EXT-key.pem -out nerveshub.DOMAIN.EXT.csr -subj '/CN=nerveshub.DOMAIN.EXT/'
  9. Create nerveshub.DOMAIN.EXT.ext with:
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keynEcipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = nerveshub.DOMAIN.EXT
  1. Generate nerveshub.DOMAIN.EXT.pem: openssl x509 -req -in nerveshub.DOMAIN.EXT.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out nerveshub.DOMAIN.EXT.pem -days 825 -sha256 -extfile nerveshub.DOMAIN.EXT.ext
  2. Move nerveshub.DOMAIN.EXT.pem y nerveshub.DOMAIN.EXT-key.pem to /etc/ssl.
  3. Modify the Caddyfileā€¦
  4. Fill the environment variables in /etc/nerves_env.
DATABASE_URL=postgresql://user:password@localhost/nerveshub

LOCAL_IPV4=127.0.0.1
ERL_COOKIE=

LIVE_VIEW_SIGNING_SALT=generated with phx.gen.secrets...
SECRET_KEY_BASE=generated with phx.gen.secrets...

HOST=nerveshub.DOMAIN.EXT
PORT=4000
URL_SCHEME=https
URL_PORT=443

DEVICE_HOST=nerveshub.DOMAIN.EXT
DEVICE_PORT=4001

API_HOST=nerveshub.DOMAIN.EXT
API_PORT=4002
API_URL_PORT=443

FROM_EMAIL=ivan@DOMAIN.EXT
NERVES_HUB_APP=all

FIRMWARE_UPLOAD_BACKEND=local
FIRMWARE_UPLOAD_PATH=
  1. Run ./bin/remote deploy and fail, it is because the systemd service file was wrong (Feedback issues with the self hosting Ā· Issue #952 Ā· nerves-hub/nerves_hub_web Ā· GitHub), so I fix it with an empty ExecStart= before the existing ExecStart=.
  2. Run ./bin/remote deploy and works!
  3. Check the NervesHub log and it shows an error because it needs the ca.pem file in /etc/ssl, so I copy the ca.pem to /etc/ssl.
  4. NervesHub web is already accesible and an API call to /api/health returns {"status": "UP"}.
  5. Run ./bin/remote iex and create an account.
  6. Download from the server the ca.pem into ~/.nerves-hub.
  7. Set the next environment variables:
    export NERVES_HUB_HOST=nerveshub.DOMAIN.EXT
    export NERVES_HUB_PORT=4001
    export NERVES_HUB_TOKEN=$token_of_my_nerveshub_account$
    export NERVES_HUB_CERT=cat /home/ivanhercaz/.nerves-hub/ca.pem
    export NERVES_HUB_ORG=organization
  8. Generate the signer certificate with mix nerves_key.signer create ivanhc, so now I have ivanhc.key and ivanhc.cert.
  9. Go to NervesHub web and create a product, then create a device and upload ivanhc.cert, copy the serial and generate the device certificate with mix nerves_hub.device cert create SERIAL --signer-cert ivanhc.cert --signer-key ivanhc.key:
    NervesHub server: nerveshub.DOMAIN.EXT:4001
    NervesHub organization: asi
    Creating certificate for SERIAL
    Signer cert path: ivanhc.cert
    Signer key path: ivanhc.key
    Finished
  10. Convert ca.pem to der with openssl x509 -outform der -in ca.pem -out ca.der.
  11. Set the configuration in config/config.exs:
cacerts =
  "/home/ivanhercaz/.nerves-hub/ca.der"
  |> File.read!()

config :nerves_hub_user_api,
  ssl: [
    cacerts: [cacerts]
  ]

config :nerves_hub_link,
  device_api_host: "nerveshub.DOMAIN.EXT",
  device_api_port: 4001,
  device_api_sni: "nerveshub.DOMAIN.EXT",
  ssl: [
    certfile:
      "/home/ivanhercaz/.nerves-hub/SERIAL-cert.pem",
    keyfile:
      "/home/ivanhercaz/.nerves-hub/SERIAL-key.pem",
    cacerts: [cacerts]
  ]
  1. And finallyā€¦ any command I use to communicate with NervesHub fails, for example, if I run mix nerves_hub.device list it returns:
22:58:44.081 [notice] TLS :client: In state :wait_cert at ssl_handshake.erl:2113 gener
ated CLIENT ALERT: Fatal - Handshake Failure
 - {:bad_cert, :hostname_check_failed}
Unhandled error: {:error, {:tls_alert, {:handshake_failure, 'TLS client: In state wait
_cert at ssl_handshake.erl:2113 generated CLIENT ALERT: Fatal - Handshake Failure\n {b
ad_cert,hostname_check_failed}'}}}

About the config/config.exs and ca.pem. My ca.pem only has one certificate inside, I mean only one pair of -----BEGIN CERTIFICATE-----CERT-----END CERTIFICATE-----. Thus I use the File.read!() and then insert into a list in :cacerts.

But based on what @jjcarstens suggested and another examples I have seen, like Poser repository, I understand the normal situation is that ca.pem has more than one. Am I doing something wrong with the ca.pem file in any of the steps I described?

1 Like
  • ca.pem can have one or many files
  • The way you are saving a reading ca.pem is producing one binary blob which is incorrect. The :cacerts needs to be a list of DER binaries or :cacertfile needs to be used to point to a file path
  • {:bad_cert, :hostname_check_failed} failed usually means the host name of the server cert (something in ca.pem) is different that what you have supplied. For the API mix tasks, you need to add:
config :nerves_hub_api,
  ssl: [
    cacerts: cacerts,
    server_name_indication: 'nerveshub.DOMAIN.EXT'
]
  • Note that the API endpoint and DEVICE endpoints might be using different ca.pem or server certificates
  • If the device is failing to connect, you need the SSL errors that are shown on the server. If there are none, the device configuration is still not right or canā€™t communicate with the host
  • Many public servers use similar trusted root CAs for their certificates. Most OS keep a store of these certificates in your /etc/ssl and lots of browsers reference that when making website connections. However, with Erlang SSL you need to be explicit. If its not in :cacerts key, it wonā€™t go looking
    • If thats a case here, using :certifi with your custom server certs might be need. Something like cacerts ++ :certifi.cacerts()

Have you gotten NervesHub to work locally? I think that you should start there and confirm this specific setup and understand before moving to a server which requires much more specific SSL configuration

Yes, I have it working locally but for rush I prefer to test the connection with the devices directly with NervesHub in a server.

I understand everything you explained and everything I have read, what I am not understanding is how my ca.pem and certificates are interacting, but after many tries I decided to leave it as it is and try another approach to use temporarily. Now I am testing Phoenix Channels and the WebSockets communication between the device and a dashboard, if not, I have a wildcard option based on MQTT communication.

When I will return to this, if the problems persists and I canā€™t solve it, I will ask again for help; if I get to solve it, I will comment here how I solve it

Thank you very much for your attention and your help, @jjcarstens.