Error when trying to set up Honeycomb/OpenTelemetry with Phoenix app

ijdickinson

I’m trying to get Honeycomb set up with my Elixir/Phoenix app. I’ve been following the guide in this blog post by Dave Lucia as a starting place.

It seemed to be going OK, but my app won’t start when I deploy it to fly.io. The error I get from flyctl logs is:

2023-03-07T15:50:11Z app[20e7bcfd] lhr [info]
{
  "Kernel pid terminated",
  application_controller,
  "{application_start_failure,
    tls_certificate_check,
    {{shutdown,
      {failed_to_start_child,tls_certificate_check_shared_state,{{badmatch,{error,enoent}
    },
    [{pubkey_os_cacerts,get,0,
      [{file,\"pubkey_os_cacerts.erl\"},{line,38}]
      },
      {tls_certificate_check_shared_state,maybe_load_authorities_trusted_by_otp,3,
      [{file,\"/app/deps/tls_certificate_check/src/tls_certificate_check_shared_state.erl\"},{line,362}]
      },
      {tls_certificate_check_shared_state,new_shared_state,2,
      [{file,\"/app/deps/tls_certificate_check/src/tls_certificate_check_shared_state.erl\"},{line,335}]
      },
      {tls_certificate_check_shared_state,handle_shared_state_initialization,2,
      [{file,\"/app/deps/tls_certificate_check/src/tls_certificate_check_shared_state.erl\"},{line,262}]
      },
      {gen_server,try_dispatch,4,
      [{file,\"gen_server.erl\"},{line,1123}]
      },
      {gen_server,handle_msg,6,
      [{file,\"gen_server.erl\"},{line,1200}]
      },
      {proc_lib,init_p_do_apply,3,
      [{file,\"proc_lib.erl\"},{line,240}]
      }
    ]
    }}},
    {tls_certificate_check_app,start,[normal,[]]}}}"
}

tls_certificate_check is not something I’ve set up, it’s added as a dependency of OpenTelemetry. So I’m not sure what it’s looking for and not finding. As far as I’ve been able to find, that’s the only diagnostic info I can locate.

Googling hasn’t given me any directions to try, so I’m feeling a bit stuck. Can anyone suggest anything to look at?

7 comments

/phoenix #flyio #honeycomb

1 1645 7

2023-03-11 02:05:09 UTC

Showing Posts 1 to 7

Show Best Posts
Show All (oldest first)
Show All (newest first)

BradS2S

enonet means it can’t find the file.

Do you have tls_certificate_check added as a dependency? I’m not familiar with this library but it is also possible you have to add a worker to the application file.

Or if that all checks out you might have a configuration issue where it is looking in the wrong place?

Post #1

ijdickinson

As far as I can tell, tls_certificate_check is only there as a transitive dependency of the Open Telemetry dependencies I have in my mix.exs. None of the docs that I’ve found mention any config for it, so I can believe that it’s failing to find a file, but I don’t know which one. When I have a moment (busy day, so far!) I’ll check the source code for that lib to see if it’s clearer what it’s looking for.

Post #2

BradS2S

It looks the file that is missing/can’t find is here:

https://github.com/erlang/otp/blob/master/lib/public_key/src/pubkey_os_cacerts.erl

You can change where it is looking for the CA cert in the config.

The option type for the library is tls_client_option() which has one option: {cacertfile, client_cafile()}

This looks like it might work:

config :tls_certificate_check, file: "/path/to/ca/file.pem"

Also from the documentation it looks like you can disable it just to see it that resolves the error.

config :tls_certificate_check, use_otp_trusted_CAs: false

Post #3

ijdickinson

Thanks @BradS2S. Looking into this some more, I can reproduce the image if I run the Docker container locally, so it’s nothing to do with Fly.io. So that eliminates one variable!

On my dev machine, there are two copies of pubkey_os_cacerts.erl: one under the asdf install tree and one in /usr/lib/erlang/lib/public_key-1.13/src/. So I’m assuming it’s installed as part of the Erlang/Elixir installation.

If I shell into the Docker container, that pubkey file is nowehere in the image. I’m starting with hexpm/elixir:1.14.2-erlang-25.1.2-debian-bullseye-20221004-slim as my base image, so I’m left with two hypotheses: either I should use a different base image for my Docker build, which does contain the missing file. Or I should add a step in the Dockerfile to either copy the missing files from my local copy (which seems like a bit of a hack, tbh) or install some additional package which will provision the pubkey_os_cacerts.erl file correctly.

Anyone have any suggestions?

(PS I haven’t tried just disabling the check. Saving that for a last resort!)

Post #4