Certifi issue / regression in Rebar3?

Questions / Help
, , ,
1

I am on this host in CircleCI, debugging a failing Elixir code. The code fails to make an API request to Stripe. Particularly, the HTTP library fails to verify the certificate of the requested host. The HTTP client in use is hackney. The error I am getting is this one:

16:25:45.892 [info] TLS :client: In state :certify at ssl_handshake.erl:1950 generated CLIENT ALERT: Fatal - Unknown CA

In pursue to answer a question “well, what would be the CAs the system knows about”? I got down to this – a call to the following function returns an empty list:

:certifi.cacerts()
# => []

After studying the contents of the function on GitHub, I see that it reads the contents of a file containing certificates. That file exists on the file system, and indeed it contains certificates:

/home/circleci/project/_build/test/lib/certifi/priv/cacerts.pem

Running a somewhat equivalent Elixir code on this host will read the certificate bytes into memory just fine:

:public_key.pem_decode(File.read!(:certifi.cacertfile())) |> Enum.map(& elem(&1, 1)) |> Enum.reverse()

I am now puzzled as to why :certify would not return a list of certificates.

I find myself unable to dig deeper, as it appears that the contents of the function is changed during the compilation time using parse_transform, specifically using using ct_expand.

Digging into parse_transform, I see that there’s a way to get details about the kind of transform performed by enabling tracing:

A debugging facility exists: passing the option {ct_expand_trace, Flags} as an option,
or adding a compiler attribute -ct_expand_trace(Flags) will enable a form of call trace.

Flags' can be ’ (no trace) or [F]', where F’ is c' (call trace), r’ (return trace), or `x’ (exception trace)'.

Would someone please assist me in understanding how to enable the tracing exactly? Do I need to modify a file in :certifi somewhere (which I can totally afford to do), or pass a compiler option in command line to iex/elixir/elixirc somehow?

I am able to reliable reproduce this very issue by entering iex -S mix shell, and calling a dummy function Stripe.PaymentIntent.create(%{}).

Trivia:

  • elixir version: 1.11.4 (released several hours ago)
  • erlang version: 23.0 (erts-11.1)
  • hackney version: 1.17.0
  • certifi version: 2.5.3
1 Like
2

Running the following code in CircleCI containers for Elixir 1.11.3 and 1.11.4 depicts the issue:

mix local.rebar --force
wget -qO- https://github.com/certifi/erlang-certifi/archive/2.5.3.tar.gz | tar xvz
cd erlang-certifi-2.5.3
~/.mix/rebar3 compile
elixir -pa _build/default/lib/certifi/ebin -e "IO.inspect(:certifi.cacerts())"

Container with elixir 1.11.3 & rebar 3.13.1:

docker run -it cimg/elixir:1.11.3 sh -c 'mix local.rebar --force; wget -qO- https://github.com/certifi/erlang-certifi/archive/2.5.3.tar.gz | tar xvz; cd erlang-certifi-2.5.3; ~/.mix/rebar3 compile; elixir -pa _build/default/lib/certifi/ebin -e "IO.inspect(:certifi.cacerts())"'
# a list of binaries, suppressed for brevity

Container with elixir 1.11.4 & rebar 3.14.4:

docker run -it cimg/elixir:1.11.4 sh -c 'mix local.rebar --force; wget -qO- https://github.com/certifi/erlang-certifi/archive/2.5.3.tar.gz | tar xvz; cd erlang-certifi-2.5.3; ~/.mix/rebar3 compile; elixir -pa _build/default/lib/certifi/ebin -e "IO.inspect(:certifi.cacerts())"'
# => []

cimg/elixir is an image maintained by CircleCI, available here: Docker Hub.

2 Likes
3

In the 1.11.4 container, upgraded parse_trans (the only dependency of certifi) to 3.4.0. No luck, same result – empty list of certificates.

1 Like
4

Possibly, related:

1 Like
5

Having a very similar issue with the latests docker image. Following.

6

Have you tried the latest version of Hackney and Certifi?

Certifi 2.6.1 was released today:

Erlang Certifi 2.6.1, providing SSL Mozilla CA bundles to your application has been released with updated certificates and compatibility with OTP >= 23.

And Hackney 1.17.2 was also released today:

  • use parse_trans 3.3.1 only (fix compatibility with Erlang < 21)
  • bump certifi version
  • Allow merging of SSL opts

From: hackney/NEWS.md at master · benoitc/hackney · GitHub

They seem potentially related. Also I would upgrade from Erlang 23.0 to 23.2 although that probably won’t help this problem.

6 Likes
7

Thanks, I had a similar issue, after upgrading Certifi and Hackney to the latest versions (2.6.1 & 1.17.2) the issue seems to be resolved!

1 Like
8

@axelson thank you! Versions you mention were not available at the time of me testing. We’ve upgraded, all good now :slight_smile:

1 Like
9

I’m glad it’s all sorted! Yeah I noticed the release announcements yesterday so I thought they might be related to the issue you ran into.