Major vulnerability fix for Rebar3

Bad news. You have to upgrade Rebar3. We just noticed that SSL validation had been partially disabled for years.

I’ve written up all the details at You've got to upgrade Rebar3

but the TL:DR; is:

  • Rebar3 didn’t properly check TLS certs for hex packages since version 3.7.0

  • Non-hex dependencies are fine

  • We don’t think there’s anybody exploiting it in the wild and it should be rather difficult

  • I’ve had time to cut releases for OTP-19 to 24 (two releases) and nightly builds are up to date

  • Older versions than 3.14 on OTP prior to 19 have no clear update path without someone having time to backport the patch further in the past.

  • it does not affect mix users

Sorry about that

26 Likes

Is it correct that we can safely run Elixir projects (with mix) with rebar3 3.14.4? I get certificate warnings when running mix local.rebar:

$ mix local.rebar

10:14:23.298 [warn]  Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'


10:14:23.518 [warn]  Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'

.asdf/installs/elixir/1.12.0-otp-24/.mix/rebar already exists, overwrite? [Yn] 

10:14:24.772 [warn]  Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'

* creating .asdf/installs/elixir/1.12.0-otp-24/.mix/rebar

10:14:24.887 [warn]  Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'


10:14:24.959 [warn]  Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'

.asdf/installs/elixir/1.12.0-otp-24/.mix/rebar3 already exists, overwrite? [Yn] 

10:14:25.356 [warn]  Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'

* creating .asdf/installs/elixir/1.12.0-otp-24/.mix/rebar3            
2 Likes

Yes: when the top-level project is managed by Mix then all Hex packages are fetched by Mix/Hex, even (transitive) dependencies that use Rebar3 as the build tool.

Mix has never verified the server’s certificate, since it does not have a CA trust store (unlike e.g. Hex and Rebar3). Instead of relying on a secure channel, it verifies the artifacts it downloads against a registry of checksums. This registry has a signature that can be verified using the public key(s) that are built into the tool, and that can be managed using mix local.public_keys.

5 Likes

The important note for any Elixir users:

If you are a mix user (with Elixir), you are not at risk: Rebar3 is used by mix only to build code, not to fetch dependencies.

Of course not a bad idea to keep rebar3 updated anyway. You can do so with mix local.rebar.

2 Likes

Note that it’s possible and even suggested to both verify artifacts and rely on a secure channel. Specifically, the secure channel prevents MITM attacks that can aim to either snoop onto the connection or pretend to be the server to return wrong information, and the artifacts validation can check for corruption or changes server-side that could point to the remote server having broken into (or the client having bad data)

Either type of protection offers an overlap with the other, but carry distinct failure modes. When put together, they require an augmented level of sophistication to successfully carry any sort of adversarial attack.
This is precisely why even though we didn’t have TLS validation on in Rebar3, we still think nothing bad happened to anyone. Defence in depth helps.

2 Likes

Right, so for fetching of packages in particular I’m quite sure the package checksums and signed registry record of Hex would have thwarted any MitM attempts. At least for Rebar3 3.8 and later (3.7, which you say was also affected by the TLS issue, used the old registry format which would allow certain substitution attacks). The main concern I would say is leakage of API credentials by rebar3_hex users.

As for Mix’s bootstrapping of Hex and Rebar3 without secure channel, this is similar to the way Rebar3 itself fetches certifi & co, relying on the Hex registry instead of the server certificate for that initial fetch, no?

1 Like

Yep. 3.7 had specific substitution attacks in limited scenarios, but those could have worked regardless iirc. API credential leakage is the trickiest and most likely issue to encounter here for sure. Generally those are at least not very frequent, on a narrow user base, and require per-device asymmetric keys with local passwords to use and aren’t very obvious to break from just the network, although user auth session (once per device) would be a likely best candidate there to then register other keys.

All of these are pretty narrow and require convoluted approaches by someone very dedicated. By comparison, what I consider to be more realistic threats of build tools are all made easier by having people just write a dependency that runs arbitrary code (as macros or configuration scripts) to exfiltrate or modify local data, and convincing people to install it.

Local passwords on publication helps protect against this becoming a worm (you require user input to do it), but there are interesting high-yield approaches then, such as lifting SSH keys (which give you potential access to git repositories and may not be password-protected) or just scanning project-parent directories that contain source code and having the ability to make changes there.

For a comparison, I’m handling all of these TLS issues very seriously, but as far as threat modelling goes, this is closing the 2nd floor window when the front door is off its hinges already, on purpose.

4 Likes

Currently mix local.rebar does not update the version for me. Perhaps a new version is not yet released?

Before:

$ ~/.asdf/installs/elixir/1.12.0-otp-24/.mix/rebar3 --version
rebar 3.14.4 on Erlang/OTP 24 Erts 12.0.1

After running mix local.rebar:

$ ~/.asdf/installs/elixir/1.12.0-otp-24/.mix/rebar3 --version
rebar 3.14.4 on Erlang/OTP 24 Erts 12.0.1

(Debug info: I’m running OTP 24.0.1, elixir 1.12.0, asdf v0.8.1)

Someone needs to update that registry I mentioned above: Mix only fetches Rebar3 versions that appear there, so it can validate its checksum, and right now that means 3.13.1 or 3.14.4 (see https://repo.hex.pm/installs/rebar3-1.x.csv).

1 Like

The primary concern at the time was malicious or compromised mirrors. In combination with a MitM vulnerability it becomes much easier to attack arbitrary users without first having to convince them to use a mirror. (And I didn’t realize at the time that Rebar3 bootstrapped certifi from Hex using registry signatures, otherwise I would have included that, as it makes for some very interesting additional threats… :slight_smile: )

Well it could be pretty good to get it updated. Do you know who to contact, and perhaps if they are already aware of it?

I’ll reach out to the Elixir core team…

3 Likes

Same issue.

What’s mix local.rebar alternative? (Manual installation?)

Our CI pipeline currently broken due to certification issue while installation.

They are just warnings, the command finishes successfully for me regardless of them.

You can always temporarily put this in your CI pipeline until the issue is resolved:

mix local.rebar --force || true

So you can override the command’s shell exit code to zero (success).

2 Likes

The command successfully finished on my local machine, but in the first steps, it’s been terminated from the side of CI.

It seems command related to CI error executed from down image layers and I should override the image in that case which I don’t intend to do it.

@voltone Any updates?

Issue on my end as well, also in alpine.

We have updated the rebar versions for those on Elixir v1.11.4+. As we said, Elixir is safe regardless as we don’t use rebar to download packages, but providing the latest rebar is a good call anyway. :slight_smile:

5 Likes

Thanks @josevalim for your input, Already, there’s a Merge Request for bitwalker/alpine-elixir-phoenix image. By releasing it, Alpine users should be happy :tada:

1 Like

The version bump to 3.15.1 is not enough, I’m afraid: the fix is part of 3.15.2.

There is now also an update for 3.13, so maybe the entry for older Elixir versions can be bumped as well…?