Installing rebar from hex.pm times out randomly when ran from within Docker container

Questions / Help
,
1

I have weird issue now with mix local.rebar --force timing out semi-randomly when I run it from within docker container.

I am attaching a Dockerfile recipe below, that fails to build on the last RUN

FROM buildpack-deps:stretch                                                                                                                                              
                                                                                                                                                                         
USER root                                                                                                                                                                
                                                                                                                                                                         
ENV LANG=C.UTF-8                                                                                                                                                         
ENV HOME=/root                                                                                                                                                           
                                                                                                                                                                         
RUN apt-get update && apt-get install -y --no-install-recommends apt-utils                                                                                               
                                                                                                                                                                         
#                                                                                                                                                                        
# ERLANG                                                                                                                                                                 
#                                                                                                                                                                        
                                                                                                                                                                         
ENV ERLANG_VERSION="1:23.2.3-1"                                                                                                                                          
                                                                                                                                                                         
RUN wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb && dpkg -i erlang-solutions_1.0_all.deb                                                      
                                                                                                                                                                         
RUN apt-get update && \                                                                                                                                                  
    apt-get install -yy --no-install-recommends esl-erlang=${ERLANG_VERSION}                                                                                             
                                                                                                                                                                         
#                                                                                                                                                                        
# ELIXIR                                                                                                                                                                 
#                                                                                                                                                                        
                                                                                                                                                                         
                                                                                                                                                                         
ENV ELIXIR_VERSION="v1.11.3"                                                                                                                                             
                                                                                                                                                                         
RUN set -xe \                                                                                                                                                            
  && ELIXIR_DOWNLOAD_URL="https://github.com/elixir-lang/elixir/releases/download/${ELIXIR_VERSION}/Precompiled.zip" \                                                   
  && buildDeps=' \                                                                                                                                                       
    unzip \                                                                                                                                                              
  ' \                                                                                                                                                                    
  && apt-get update \                                                                                                                                                    
  && apt-get install -y --no-install-recommends $buildDeps \                                                                                                             
  && curl -fSL -o elixir-precompiled.zip $ELIXIR_DOWNLOAD_URL \                                                                                                          
  && unzip -d /usr/local elixir-precompiled.zip \                                                                                                                        
  && rm elixir-precompiled.zip                                                                                                                                           
                                                                                                                                                                         
#                                                                                                                                                                        
# Hex  + Rebar                                                                                                                                                           
#                                                                                                                                                                        
                                                                                                                                                                         
ENV MIX_ENV=prod                                                                                                                                                         
                                                                                                                                                                         
RUN mix local.hex --force                                                                                                                                                
RUN mix local.rebar --force

This last step times out when building the dockedr image. I can also reproduce the same issue by running docker run ... mix local.rebar --force. Every 5 or 6 runs this fails with timeout:

** (Mix) request timed out after 60000ms

Could not install Rebar because Mix could not download metadata at https://repo.hex.pm/installs/rebar3-1.x.csv.

or

** (Mix) request timed out after 60000ms
Could not fetch rebar at:
    https://repo.hex.pm/installs/1.0.0/rebar-2.6.2
Please download the file above manually to your current directory and run:
    mix local.rebar rebar ./rebar

When I run mix local.rebar --force on the host machine I have no such issue no matter how many times I run it.

There are obvious differences in the set up: my dev machine is Ubuntu 20.4, and Docker image is Debian stretch. Again, I can’t reproduce it on my dev machine’s host OS.

When the above sometimes succeeds in Docker image, then I get errors when installing deps:

$ mix deps.get
Failed to fetch record for 'hexpm/timex' from registry (using cache instead)00:16
** (Mix) Unknown package timex in lockfile

Again, from CI / Docker image only. Not from host machine, that works fine.

So… any ideas anyone what can cause the Docker image to have these connectivity issues? This is happening both on CI and on my dev machine when I run the image.

2

Semaphore CI (which we use) published this:

[Network timeouts towards hex.pm](https://status.semaphoreci.com/incidents/vp192wyf41xl)[Subscribe](https://status.semaphoreci.com/#subscribe-modal-vp192wyf41xl)

**Investigating** - We are seeing sporadic timeouts when attempting to reach hex.pm. Our team is actively working with the upstream provider to pinpoint and resolve the issue.

In the meantime, following workarounds are working:
- Use hex mirror - `HEX_MIRROR=[https://hexpm.upyun.com](https://hexpm.upyun.com/) mix deps.get`
- Add retry to `mix deps.get` as follows: `retry -t 10 mix deps.get`
<small>Apr 23, 09:00 UTC</small>

on their status page https://status.semaphoreci.com/

So I guess there’s some hex.pm issues at the moment. // cc @wojtekmach ?

Doesn’t really explain why in Docker image on my home workstation I get the same network issues, but not on the host OS on the same machine.

3

We are aware of the issue and working with Fastly and Semaphore CI who are working with their upstream providers to resolve the issue.

As a disclaimer to the suggested workaround of using a mirror is that we cannot vouch for our guarantee the security of a non-official mirror when authenticating against the repository or when using private packages. Using a mirror is at your own risk.

1 Like
4

Yeah, I am definitely not using this one.

Does hex.pm have a status page? Maybe we should get one up?

5

We have one now: https://status.hex.pm/.

3 Likes
6

Perfect, thank you for setting it up!

7

Having the same issue.

From Gitlab CI, I get the timeout error.

Would if be possibel to have hex and bar installed in the elixir docker image by default?

@ericmj

8

So I got failures with both rebar3 and hex.

I’ve changed this:

RUN mix local.hex --force && mix local.rebar

for this:

RUN wget https://s3.amazonaws.com/rebar3/rebar3 && chmod +x rebar3 && mv rebar3 ~/.mix
RUN wget https://github.com/rebar/rebar/wiki/rebar && chmod +x rebar && mv rebar ~/.mix
RUN mix archive.install github hexpm/hex branch latest --force

and now I got it working.

9

I just ran into this issue myself on GitLab. It’s not hex or rebar but GitLab is having network issues for its runners. 2021-09-26: Intermittent networking issues with some shared runner jobs (#5590) · Issues · GitLab.com / GitLab Infrastructure Team / production · GitLab