mikl
December 17, 2024, 4:04am
1
We’ve recently started seeing our CI jobs failing due to mix deps.get hanging. It stalls for an hour, then the CI job gets a timeout, our script looks like this:
$ mix local.hex --force --if-missing
$ mix local.rebar --force --if-missing
$ mix hex.repo add oban https://getoban.pro/repo --fetch-public-key ${OBAN_KEY_FINGERPRINT} --auth-key ${OBAN_LICENSE_KEY}
$ mix deps.get
This is with Gitlab CI, using the hexpm/elixir:1.17.3-erlang-26.2.5.4-debian-bookworm-20241016-slim docker image.
Is there any way to see where it is getting stuck? mix help deps.get does not list any verbose or debugging flags.
Can you curl the package from within CI? Maybe try that just to verify the credentials in the url are correct? Something like:
curl -I https://getoban.pro/repo --fetch-public-key ${OBAN_KEY_FINGERPRINT} --auth-key ${OBAN_LICENSE_KEY}
1 Like
mikl
December 17, 2024, 9:01pm
3
It works about half the time, so the credentials must be right. And if they were wrong, Hex would throw an error message, not just hanging.
Authenticating and fetching from the getoban.pro repo shouldn’t time out. However, we’ve heard about network trouble from CI instances in the past.
Try switching to the newer, globally distributed repo instead: https://repo.oban.pro. That will be served closer to your CI servers and should have better connectivity guarantees.
4 Likes
mikl
December 23, 2024, 10:42am
5
I’m far from certain that it’s the Oban repo causing mix deps.get to hang, changing the URL didn’t help at least. Wish I could find a way to see what is causing it to hang.
1 Like
axelson
December 23, 2024, 6:10pm
6
What does MIX_DEBUG=true mix deps.get show? It should print output kind of like:
> MIX_DEBUG=true mix deps.get
-> Running mix loadconfig (inside Rewrite.MixProject)
<- Ran mix loadconfig in 0ms
-> Running mix deps.get (inside Rewrite.MixProject)
-> Running mix archive.check (inside Rewrite.MixProject)
<- Ran mix archive.check in 0ms
Resolving Hex dependencies...
Resolution completed in 0.038s
Unchanged:
benchee 1.3.1
benchee_dsl 0.5.3
bunt 1.0.0
credo 1.7.8
deep_merge 1.0.0
dialyxir 1.4.4
earmark_parser 1.4.41
erlex 0.2.7
ex_doc 0.34.2
excoveralls 0.18.3
file_system 1.0.1
fss 0.1.1
glob_ex 0.1.10
jason 1.4.4
kino 0.14.2
makeup 1.1.2
makeup_elixir 0.16.2
makeup_erlang 1.0.1
nimble_parsec 1.4.0
sourceror 1.7.0
statistex 1.0.0
table 0.1.2
text_diff 0.1.0
All dependencies are up to date
<- Ran mix deps.get in 469ms
Another thing you could check is MIX_PROFILE=deps.get mix deps.get. Although I’m not sure if that’ll help if mix deps.get hangs forever.
Example output
> MIX_PROFILE=deps.get mix deps.get
-> Profiling mix deps.get (inside Rewrite.MixProject)
Resolving Hex dependencies...
Resolution completed in 0.064s
Unchanged:
benchee 1.3.1
benchee_dsl 0.5.3
bunt 1.0.0
credo 1.7.8
deep_merge 1.0.0
dialyxir 1.4.4
earmark_parser 1.4.41
erlex 0.2.7
ex_doc 0.34.2
excoveralls 0.18.3
file_system 1.0.1
fss 0.1.1
glob_ex 0.1.10
jason 1.4.4
kino 0.14.2
makeup 1.1.2
makeup_elixir 0.16.2
makeup_erlang 1.0.1
nimble_parsec 1.4.0
sourceror 1.7.0
statistex 1.0.0
table 0.1.2
text_diff 0.1.0
All dependencies are up to date
Profile results of #PID<0.114.0>
# CALLS % TIME µS/CALL
Total 67945 100.0 84998 0.13
:digraph.delete/1 2 0.00 0 0.00
:digraph.set_type/2 2 0.00 0 0.00
:digraph.check_type/3 2 0.00 0 0.00
:digraph.new/0 2 0.00 0 0.00
Hex.Registry.Server.open/1 1 0.00 0 0.00
Hex.Registry.Server.open/0 1 0.00 0 0.00
:digraph_utils.forest/3 2 0.00 0 0.00
:erlang.setelement/3 3 0.00 0 0.00
:erlang.monotonic_time/1 2 0.00 0 0.00
:erlang.atom_to_list/1 1 0.00 0 0.00
List.flatten/1 1 0.00 0 0.00
Path.relative_to_cwd/1 8 0.00 0 0.00
Path.dirname/1 1 0.00 0 0.00
Hex.Solver.Constraints.Range.include/1 5 0.00 0 0.00
:eval_bits.expr_grp/3 1 0.00 0 0.00
Map.new/1 1 0.00 0 0.00
Map.merge/3 1 0.00 0 0.00
Map.drop_keys/2 3 0.00 0 0.00
anonymous fn/1 in Hex.RemoteConverger.packages_from_requests/1 8 0.00 0 0.00
Hex.RemoteConverger."-fun.request_to_dependency/1-"/1 8 0.00 0 0.00
anonymous fn/1 in Hex.RemoteConverger.verify_input/2 8 0.00 0 0.00
Hex.RemoteConverger."-fun.verify_repo/1-"/1 1 0.00 0 0.00
Hex.RemoteConverger.with_children/2 1 0.00 0 0.00
Hex.RemoteConverger.verify_resolved/2 1 0.00 0 0.00
Hex.RemoteConverger.verify_prefetches/1 1 0.00 0 0.00
Hex.RemoteConverger.verify_otp_app_names/1 1 0.00 0 0.00
Hex.RemoteConverger.verify_input/2 1 0.00 0 0.00
Hex.RemoteConverger.verify_deps/2 1 0.00 0 0.00
Hex.RemoteConverger.unlock_deps/2 1 0.00 0 0.00
Hex.RemoteConverger.print_success/2 1 0.00 0 0.00
Hex.RemoteConverger.print_dependency_group/2 4 0.00 0 0.00
Hex.RemoteConverger.print_category/1 1 0.00 0 0.00
Hex.RemoteConverger.packages_from_requests/1 1 0.00 0 0.00
Hex.RemoteConverger.normalize_resolved/1 1 0.00 0 0.00
Hex.RemoteConverger.lock_merge/2 1 0.00 0 0.00
Hex.RemoteConverger.group_dependency_changes/2 1 0.00 0 0.00
Hex.RemoteConverger.do_with_children/2 1 0.00 0 0.00
Hex.RemoteConverger.dep_info_from_lock/1 1 0.00 0 0.00
Hex.RemoteConverger.add_apps_to_resolved/2 1 0.00 0 0.00
Kernel.__info__/1 6 0.00 0 0.00
System.normalize_time_unit/1 2 0.00 0 0.00
:lists.mergel/2 4 0.00 0 0.00
:lists.do_flatten/2 5 0.00 0 0.00
:lists.flatten/1 1 0.00 0 0.00
:erl_eval.exprs/6 3 0.00 0 0.00
:erl_eval.empty_fun_used_vars/0 3 0.00 0 0.00
Mix.Dep.Umbrella.unloaded/0 1 0.00 0 0.00
:logger_server.check_level/1 2 0.00 0 0.00
<MANY MORE LINES>
:erlang.binary_to_atom/2 7370 1.59 1352 0.18
:erl_scan.scan_string_no_col/5 24921 1.63 1389 0.06
:gen.do_call/4 2194 1.83 1554 0.71
:io_lib_format.build_small/1 17526 2.22 1887 0.11
:erlang.atom_to_binary/1 15111 2.22 1888 0.12
:io_lib_format.collect/2 17526 2.44 2075 0.12
:io_lib_format.build_limited/5 17526 2.62 2228 0.13
:elixir_aliases.do_concat/2 14316 2.77 2357 0.16
:erts_internal.prepare_loading/2 38 16.22 13784 362.74
Profile done over 1270 matching functions
It was a little tricky to find, but here is where that is documented: Mix — Mix v1.18.0
mix help won’t print that out as help textmix has a manpage but it isn’t installed on my system (where I use asdf)
Searching hexdocs.pm for MIX_DEBUG doesn’t yield any results because it actually seems to search for all mentions of mix or debug and there are a lot of those
I found the docs because I already knew there was a MIX_DEBUG environment variable, so I downloaded Elixir’s source code and searched via git grep.
One last thing. If you have the dependencies, i.e. you’ve cached the deps/, ~/.mix, ~/.hex then you can add HEX_OFFLINE=true to skip contracting hex.pm (that’s documented at mix hex.config — Hex v2.0.6 )
3 Likes
