Having issues loading the `host_key` for BEAM ssh daemon

Questions / Help
1

I’m a huge fan of the remote shell capabilities that elixir 1.9 added.

I’m working in an environment where I won’t be able to connect via BEAM node networking and would like to use the beam :ssh module to give a similar experience of elixir’s releases remote to my team.

I’m following the Erlang based examples here but I’m running into issues.
https://www.erlang-solutions.com/blog/secure-shell-for-your-erlang-node.html
http://erlang.org/doc/apps/ssh/users_guide.html

I’m running these on my Mac Mojave (10.14.6) with these Erlang/Elixir versions

Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]
Interactive Elixir (1.10.2) - press Ctrl+C to exit (type h() ENTER for help)

And this openssh version

➜  ssh -V
OpenSSH_7.9p1, LibreSSL 2.7.3

First I generate the keys (with no password)

mkdir ssh_dir
ssh-keygen -t rsa -f ssh_dir/ssh_host_rsa_key

Then I try to run this elixir code and its telling me that there is no host key available

iex(4)> :ok = :ssh.start()
:ok
iex(5)> :ssh.daemon(8989, system_dir: 'ssh_dir', password: 'foo')
{:error, 'No host key available'}

Then I try it in Erlang and get the same answer

2> ok = ssh:start().
ok
3> ssh:daemon(8989, [{system_dir, "ssh_dir"}, {password, "foo"}]).
{error,"No host key available"}

I was wondering if anyone had any insight on how to debug this issue.

(Here are the key files I’m using to debug. I am NOT using these keys anywhere, just to test the ssh module)

➜  cat ssh_dir/ssh_host_rsa_key.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTprdAtxwCexonmzP3qDvMvukg+n/cNr86mE0CkcnAzCjmfPZBW1ywy8RDfz06uXmUq02OlmioZ2OUR7fTK+ZViVkjrwyW232EoVNW2FnwHS//LPQsRuY3JZU9KCKhhRgNUK3elCY4jOARCuVi8dw3+vqnTkBNYkNXvJNkHcwcWwwja9kHi/ZbXpKuZCTaM1CGUB8cOrc90d5hdfNdYyh4yQf2+Zu3jOJPqHOqHD3T/Bw+a56fKx7wi4+EqAHVv42Pfk8FB05CFhvGHcn+0ImhCtjUtSyTLXuXtrAkWbvPnpDVyfzPsmzMqX3Pj/lYdZEKQ07OVNb94N3hCVFhyBEb grantpowell@Grants-MacBook-Pro-2.local

➜  cat ssh_dir/ssh_host_rsa_key
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcn
NhAAAAAwEAAQAAAQEA06a3QLccAnsaJ5sz96g7zL7pIPp/3Da/OphNApHJwMwo5nz2QVtc
sMvEQ389Orl5lKtNjpZoqGdjlEe30yvmVYlZI68Mltt9hKFTVthZ8B0v/yz0LEbmNyWVPS
gioYUYDVCt3pQmOIzgEQrlYvHcN/r6p05ATWJDV7yTZB3MHFsMI2vZB4v2W16SrmQk2jNQ
hlAfHDq3PdHeYXXzXWMoeMkH9vmbt4ziT6hzqhw90/wcPmuenyse8IuPhKgB1b+Nj35PBQ
dOQhYbxh3J/tCJoQrY1LUsky17l7awJFm7z56Q1cn8z7JszKl9z4/5WHWRCkNOzlTW/eDd
4QlRYcgRGwAAA+CowULCqMFCwgAAAAdzc2gtcnNhAAABAQDTprdAtxwCexonmzP3qDvMvu
kg+n/cNr86mE0CkcnAzCjmfPZBW1ywy8RDfz06uXmUq02OlmioZ2OUR7fTK+ZViVkjrwyW
232EoVNW2FnwHS//LPQsRuY3JZU9KCKhhRgNUK3elCY4jOARCuVi8dw3+vqnTkBNYkNXvJ
NkHcwcWwwja9kHi/ZbXpKuZCTaM1CGUB8cOrc90d5hdfNdYyh4yQf2+Zu3jOJPqHOqHD3T
/Bw+a56fKx7wi4+EqAHVv42Pfk8FB05CFhvGHcn+0ImhCtjUtSyTLXuXtrAkWbvPnpDVyf
zPsmzMqX3Pj/lYdZEKQ07OVNb94N3hCVFhyBEbAAAAAwEAAQAAAQB+X44NowejZ5NvUJlP
53CFSPx0B34uo7W8q9FXlmrrpjtJnBWBYqbOnWw7jQ6nMLWqkjoFyEMHQmxPzh6xhQXLzu
2dg9i1fqrMdXMIaLgSyaqt506H3XF6PgOE96Dt5kU9Ng4K11SW8W6cXnAmixkdh8r/+JYm
XmStT1iGL9B+oLvJWXqB+nSfYI1/25/SRJldmPmk02cxyM4f4nyKMjCX+/Fyd0GPTOVkf6
fnzS9eC810HP5bmYEHgZYfJYQAqCMk0hkOIp5RH5jimu6Csv1uMrptE/7dkYgXzU+QJDp3
SbzmYddfaq+9fjwpFftyHXQSk+MtA7sMpD4n2+bAvAx5AAAAgQDkvvedNDP8OVvkRia/CT
x8hzqgjrlv18UV21zVU8maeoLlsjSuUD0f41VRbD5iz1zOGOiQQX05feLOVBdZtggqA+Mj
ZJdDM7r0r9fmPz9oRKEbM7VN8TNnvkS9idA7ptvvjnSC5KRyNAPwDVgd3HqC3d2UW1L1yL
00t5iRa3ewlQAAAIEA+f+XBI6iR470xytUIlM5jFFZxwH/xkAhNNSwhAONM4AkZ/5ImuJg
ZMxSASw2/Q6UCR1/Nsp32FAMtEOdmKMfKVlhFwJ2Y8C/qzN7OZ1P5IxaGBeiphDScCTG9i
otP+5bRnff3Lzfm1eIeiXkAvHGKlnGxmQx2woS2lnAldZi/NcAAACBANi7dN74LVS9m3av
Hc2Cs22JFdRv2MUjS1NJlF9t8QXoXolH2mUnEK6EfSDRHgmelZ4kkIMEHKwHpCfTKwJmci
B3yfpB4gdnS0QGr+BftpcjaMCv/5lzlmP86e8ucLAq87lSD9lK5Ux2IUxCOFpj7nxS8oc+
g60pv8yPkpbWyaFdAAAAJmdyYW50cG93ZWxsQEdyYW50cy1NYWNCb29rLVByby0yLmxvY2
FsAQIDBA==
-----END OPENSSH PRIVATE KEY-----

AGAIN, these are not keys I’m really using, they’re just for debugging why the BEAM won’t pick up the host key

Any insights would be appreciated

2

Update: figured out I can make changes in the OTP source, and then compile them into my node (do at your own risk!). Used io:format to trace down the line causing issues that was being swallowed by a try catch.

https://github.com/erlang/otp/blob/OTP-22.0.7/lib/ssh/src/ssh_file.erl#L149

Here is minimal example that replicates the issue, along with the binary data to replicate at home

pem = File.read!("ssh_dir/ssh_host_rsa_key")
[{_, bin, :not_encrypted} = entry] = :public_key.pem_decode(pem)
bin |> Base.encode64 |> IO.inspect()
:public_key.pem_entry_decode(entry)
# iex(57)> :public_key.pem_entry_decode(entry)
# ** (CaseClauseError) no case clause matching: {"ssh-rsa", <<1, 0, 1>>}
#     pubkey_ssh.erl:204: :pubkey_ssh.new_openssh_decode/6
"b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcnNhAAAAAwEAAQAAAQEAvqxCT7ag3FztOXcGECjhgPoabAfJzrDD5FotaHcr/tQkkg2B9EsoYlEh1a4OCaT+0YQ2LhIxaItRP1wfJeatvJW5lv2aSFxJHsl3I+PGQeH4R27rLvv41H7D51nPh7d7eqwCixQkSlyaCjUmSqB1usCZYRrbaA8HX01GEcwRrDXG+m/oMeCPZPzVluw88/WH8yRldNkMiuuSmF1/oOVFIUfOcXjL4jLyOW3rSSFzlO9qS17BdU6FDVwObEScaQYAjoKYuoUxshiduA/PM9fE/KCwpoxNDHxyZiX7bfYecLrT3QjY3hhpuEKcfW8AVPpYQaCALUMgmPnh14qLeoY0gQAAA+A6tCZQOrQmUAAAAAdzc2gtcnNhAAABAQC+rEJPtqDcXO05dwYQKOGA+hpsB8nOsMPkWi1odyv+1CSSDYH0SyhiUSHVrg4JpP7RhDYuEjFoi1E/XB8l5q28lbmW/ZpIXEkeyXcj48ZB4fhHbusu+/jUfsPnWc+Ht3t6rAKLFCRKXJoKNSZKoHW6wJlhGttoDwdfTUYRzBGsNcb6b+gx4I9k/NWW7Dzz9YfzJGV02QyK65KYXX+g5UUhR85xeMviMvI5betJIXOU72pLXsF1ToUNXA5sRJxpBgCOgpi6hTGyGJ24D88z18T8oLCmjE0MfHJmJftt9h5wutPdCNjeGGm4Qpx9bwBU+lhBoIAtQyCY+eHXiot6hjSBAAAAAwEAAQAAAQEAvZfvv++q+DN2tyNA5iSgRm1qhG0iH+Hoh1GbwTpuhs9PJeNEizOIjmp6KE0QLEriDldWwGzasEeHnGbvmfJMYy7GAftdxAHvuys8ynUcmDondHA4btmhzTnXdl35W8v4fjw3sy9t/nshuHCtCJuRNQd5+/KGpGv9OqhNujieggqmusuTinb5daRmwgswJE4OjNwNbJi4fvKWMF6ccVkqVW59FceG9Ps0XBURdz1SxybQt1DqjmViYwcerIXPAW9OD3GFzLqNeWrUZ2s8C87685VA9GfD1LEn1rVh4o3rjpTBaPjDu95KKWZrgeFDzOz8A0Fx/U6R9ladOzHtqGWfmQAAAIAEhJox7yRqbG8n6dYIkfZsKAEqZRdq72yv+hepHtLiImIuKJn7lSejSgLjHXyRqBEF3IRV2MQ4gob83dCJBcvz1qGbrgDcHdrfpeKFZ4QN+6hcr1Omm4GpDS4ClNdtkud5wjoYyRzjP3hpuXw9thmuonEGu+ezQj34hg3lothN9AAAAIEA7S7pKOwkpZj+SVh5nxpRslORnLXMYwaoI8SQ75Pvg7oSlxvk5UGZvpZLev08W0gdKSMwzumoOG5CBbXz/+WzVmJNQb1q/4/p3B54tOsYOIRRXUPBKnbrjJ6D297RJr3LFf4V9CsM1yhwTQZbldl4Q6hfhWNR7jrvhFaNGIDD/nsAAACBAM3MvjMVExHXQPXUm7MKxqcXZ1+4mzRxYxFp1JrcoEoyUfRagGe2qO0FyZhXdwczokHrJhtXTLJVsm9G6R1mXH7i5czJGrClD5zEGulS8UGNd7MznjtzDAUzLlegPd+VlQC73ADpL8zkcAaBU3VdC0PvbhK3xmE21yHjA/eLCOYzAAAAJmdyYW50cG93ZWxsQEdyYW50cy1NYWNCb29rLVByby0yLmxvY2FsAQIDBA=="
3

continued from my above comments, here is another example with the line numbers in otp giving errors. I’ve included the key in question binary so you can reproduce at home!

key_bin = Base.decode64!("b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcnNhAAAAAwEAAQAAAQEAvqxCT7ag3FztOXcGECjhgPoabAfJzrDD5FotaHcr/tQkkg2B9EsoYlEh1a4OCaT+0YQ2LhIxaItRP1wfJeatvJW5lv2aSFxJHsl3I+PGQeH4R27rLvv41H7D51nPh7d7eqwCixQkSlyaCjUmSqB1usCZYRrbaA8HX01GEcwRrDXG+m/oMeCPZPzVluw88/WH8yRldNkMiuuSmF1/oOVFIUfOcXjL4jLyOW3rSSFzlO9qS17BdU6FDVwObEScaQYAjoKYuoUxshiduA/PM9fE/KCwpoxNDHxyZiX7bfYecLrT3QjY3hhpuEKcfW8AVPpYQaCALUMgmPnh14qLeoY0gQAAA+A6tCZQOrQmUAAAAAdzc2gtcnNhAAABAQC+rEJPtqDcXO05dwYQKOGA+hpsB8nOsMPkWi1odyv+1CSSDYH0SyhiUSHVrg4JpP7RhDYuEjFoi1E/XB8l5q28lbmW/ZpIXEkeyXcj48ZB4fhHbusu+/jUfsPnWc+Ht3t6rAKLFCRKXJoKNSZKoHW6wJlhGttoDwdfTUYRzBGsNcb6b+gx4I9k/NWW7Dzz9YfzJGV02QyK65KYXX+g5UUhR85xeMviMvI5betJIXOU72pLXsF1ToUNXA5sRJxpBgCOgpi6hTGyGJ24D88z18T8oLCmjE0MfHJmJftt9h5wutPdCNjeGGm4Qpx9bwBU+lhBoIAtQyCY+eHXiot6hjSBAAAAAwEAAQAAAQEAvZfvv++q+DN2tyNA5iSgRm1qhG0iH+Hoh1GbwTpuhs9PJeNEizOIjmp6KE0QLEriDldWwGzasEeHnGbvmfJMYy7GAftdxAHvuys8ynUcmDondHA4btmhzTnXdl35W8v4fjw3sy9t/nshuHCtCJuRNQd5+/KGpGv9OqhNujieggqmusuTinb5daRmwgswJE4OjNwNbJi4fvKWMF6ccVkqVW59FceG9Ps0XBURdz1SxybQt1DqjmViYwcerIXPAW9OD3GFzLqNeWrUZ2s8C87685VA9GfD1LEn1rVh4o3rjpTBaPjDu95KKWZrgeFDzOz8A0Fx/U6R9ladOzHtqGWfmQAAAIAEhJox7yRqbG8n6dYIkfZsKAEqZRdq72yv+hepHtLiImIuKJn7lSejSgLjHXyRqBEF3IRV2MQ4gob83dCJBcvz1qGbrgDcHdrfpeKFZ4QN+6hcr1Omm4GpDS4ClNdtkud5wjoYyRzjP3hpuXw9thmuonEGu+ezQj34hg3lothN9AAAAIEA7S7pKOwkpZj+SVh5nxpRslORnLXMYwaoI8SQ75Pvg7oSlxvk5UGZvpZLev08W0gdKSMwzumoOG5CBbXz/+WzVmJNQb1q/4/p3B54tOsYOIRRXUPBKnbrjJ6D297RJr3LFf4V9CsM1yhwTQZbldl4Q6hfhWNR7jrvhFaNGIDD/nsAAACBAM3MvjMVExHXQPXUm7MKxqcXZ1+4mzRxYxFp1JrcoEoyUfRagGe2qO0FyZhXdwczokHrJhtXTLJVsm9G6R1mXH7i5czJGrClD5zEGulS8UGNd7MznjtzDAUzLlegPd+VlQC73ADpL8zkcAaBU3VdC0PvbhK3xmE21yHjA/eLCOYzAAAAJmdyYW50cG93ZWxsQEdyYW50cy1NYWNCb29rLVByby0yLmxvY2FsAQIDBA==")

:pubkey_ssh.decode(key_bin, :new_openssh)
# ** (CaseClauseError) no case clause matching: {"ssh-rsa", <<1, 0, 1>>}
#     /Users/grantpowell/.asdf/installs/erlang/22.0.7/lib/public_key-1.6.7/src/pubkey_ssh.erl:204: :pubkey_ssh.new_openssh_decode/6

It looks like I’m getting {ssh-rsa, <<1,0,1>>} on these lines https://github.com/erlang/otp/blob/OTP-22.0.7/lib/public_key/src/pubkey_ssh.erl#L204-L213. My new theory is that this version of otp doesn’t support the openssh rsa key I’m using

4

Update, the :ssh.daemon works with openssh ed25519 keys

mkdir ssh_dir
ssh-keygen -t ed25519 -f ssh_dir/ssh_host_ed25519_key

:ok = :ssh.start()
{:ok, sshd} = :ssh.daemon(8989, system_dir: 'ssh_dir', password: 'foo')

➜  erl_ssh_2 ssh 127.0.0.1 -p 8989
SSH server
Enter password for "grantpowell"
password:
Eshell V10.3.5.2  (abort with ^G)
1>
4 Likes