Mix deps.get crashes with a timeout

This strange one started today…

Anyone have an idea what’s wrong?

C:\Elixir\BotEngine
λ mix deps.get
** (exit) exited in: GenServer.call(#PID<0.115.0>, {:versions, "certifi"}, 60000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:737: GenServer.call/3
    (hex) lib/hex/remote_converger.ex:208: Hex.RemoteConverger.check_dep/2
    (elixir) lib/enum.ex:651: anonymous fn/3 in Enum.each/2
    (stdlib) lists.erl:1263: :lists.foldl/3
    (elixir) lib/enum.ex:1772: Enum.each/2
    (hex) lib/hex/remote_converger.ex:38: Hex.RemoteConverger.converge/2
    (mix) lib/mix/dep/converger.ex:91: Mix.Dep.Converger.all/4
    (mix) lib/mix/dep/converger.ex:50: Mix.Dep.Converger.converge/4


C:\Elixir\BotEngine
λ
1 Like

Amazon S3 which Hex uses to host all packages is having an outage: https://status.aws.amazon.com/. Until it’s back up the package repository will be unavailable.

2 Likes

Would be good if we can get an “outage error” in mix … that would give us less head scratching

1 Like

We don’t know if the error is due to an outage so giving an “outage error” in mix would be hard. I’m currently investigating why Hex is timing out internally instead of showing an error message. You can set HEX_OFFLINE=1 to force Hex into offline mode and use the local cache when available.

2 Likes

Thanks. maybe hex should report this to the user, then prompt to switch to hex_offline_mode

1 Like

It’s supposed to do that, but there is a bug preventing that from happening. That’s what I’m currently investigating.

2 Likes

Even when specified it does not work once you’ve ran it normally… Why is certifi not cached locally when I’ve not cleaned my deps?!? My build is hosed right now at work, blah…

$ HEX_OFFLINE=1 mix compile

Unchecked dependencies for environment dev:
* plug (Hex package)
  the dependency is not available, run "mix deps.get"
** (Mix) Can't continue due to errors on dependencies

$ HEX_OFFLINE=1 mix deps.get

** (Mix) Hex is running in offline mode and the registry entry for package certifi is not cached locally

$ mix deps.get

** (exit) exited in: GenServer.call(#PID<0.118.0>, {:versions, "certifi"}, 60000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:737: GenServer.call/3
    (hex) lib/hex/remote_converger.ex:208: Hex.RemoteConverger.check_dep/2
    (elixir) lib/enum.ex:651: anonymous fn/3 in Enum.each/2
    (stdlib) lists.erl:1263: :lists.foldl/3
    (elixir) lib/enum.ex:1772: Enum.each/2
    (hex) lib/hex/remote_converger.ex:38: Hex.RemoteConverger.converge/2
    (mix) lib/mix/dep/converger.ex:91: Mix.Dep.Converger.all/4
    (mix) lib/mix/dep/converger.ex:50: Mix.Dep.Converger.converge/4

I’m guessing by registry you do not mean a global one like maven uses (I’ve not looked into how hex stores things locally beyond it ending up in deps) so… my work is stuck… >.<

Hmm, I could switch everything to git I guess… >.>

/me really dislikes AWS…

1 Like

Some of your dependencies are not available so if they are not cached locally Hex will not be able to install them in your project.

1 Like

Where would they be locally cached, what is the cache directory (for maven it is in $HOME/.maven on my server for example)?

1 Like

By default it’s ~/.hex/packages

3 Likes

To elaborate on @wojtekmach’s response to cache ~/.hex/packages you should also cache ~/.hex/cache.ets for Hex to work without being able to connect to the repository.

2 Likes

Awesome, I think I’ll cache that on my CI docker image to keep from downloading deps every time. ^.^

Thanks!

2 Likes

Looks like this is resolved now. Phew!

Perhaps a good time to plan out a better strategy to handle cases like this from happening in the future.

2 Likes

We have some ideas for making the repository more resilient for future incidents.

As a first step we will add S3 replication to buckets in Europe and Asia and set Fastly (our CDN in front of S3) to fail over to the other regions if the main origin start erroring out.

Another thing would be for the community to help out with providing mirrors. We have some unofficial mirrors already [1] but AFAIK they are all basically caching HTTP proxies so they may not help if the origin goes down. We have specified in the docs everything needed to run a mirror [2] so if anyone wants to create and host a mirror that would be really helpful for the future.

[1] https://hex.pm/docs/mirrors
[2] https://github.com/hexpm/specifications/blob/master/endpoints.md#endpoints-1

6 Likes

I have a fairly significant amount of space and bandwidth available if I can help there. :slight_smile:

2 Likes

That’s great, thank you for that.

We also need to build the software that does the mirroring.

1 Like

Whenever that is built then feel free to hit me up and I’ll set up something so you can mirror it over. :slight_smile:

1 Like

Hello folks,

I am still having an issue particularly with package certifi. AWS statuses looks fine yet I have the following error when installing my deps :

* Getting certifi (Hex package)
  Checking package (https://repo.hex.pm/tarballs/certifi-1.0.0.tar)
Request failed (:timeout)
** (Mix) Package fetch failed and no cached copy available

What is strange is that other packages are downloaded well but only the certifi one fails from time to time and this often break my CI.

Certifi is a very big package so it’s likely it’s the only that breaks because the other’s are much smaller. Are you using latest version of Hex and how long does it take until the request times out?

We have the following versions?

$ mix hex.info
Hex:    0.16.0
Elixir: 1.4.2
OTP:    19.3.1

Should I try with latest version of elixir and hex? The timeout is really small but I can’t find a way to make it longer (it’s less than 10sec I think)