Having trouble with a Hex Error - GenServer Hex.Registry.Server terminating

I started an error yesterday during deploys that I’m having trouble tracking down. These are automated deploys that have been running without issue for months and no changes were made to the CI or environment, that I’m aware of. The error doesn’t occur every time, but has been happening fairly regularly (maybe 50% of the time). Any help would be greatly appreciated.

I’m running the following inside a elixir:1.8.0 container.

export MIX_ENV=test
mix local.hex --force
mix local.rebar --force
mix deps.get
mix compile

mix test --trace

The error seems to be occurring as a result of mix local.hex --force

16:57:11.399 [error] GenServer Hex.Registry.Server terminating
** (stop) :data_error
    (hex) lib/hex/registry/server.ex:355: Hex.Registry.Server.write_result/4
    (hex) lib/hex/registry/server.ex:228: Hex.Registry.Server.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:get_package, "hexpm", "scrivener", {:ok, {200, "", %{'accept-ranges' => 'bytes', 'age' => '498277', 'cache-control' => 'public, max-age=3600', 'connection' => 'close', 'content-length' => '1814', 'content-type' => 'application/octet-stream', 'date' => 'Wed, 08 May 2019 16:57:11 GMT', 'etag' => '"9b42f93feeaba19c5117a66cc1849f37"', 'last-modified' => 'Fri, 01 Mar 2019 17:17:18 GMT', 'server' => 'AmazonS3', 'via' => '1.1 varnish', 'x-amz-id-2' => 'rN8T2l+a8x7GsDz2x4jlcA7SF1h+gc6LuSejtiwRO4Un+AIMksOwBcaxBxEzwa0/ih3bv+6C0oY=', 'x-amz-meta-surrogate-control' => 'public, max-age=604800', 'x-amz-meta-surrogate-key' => 'registry registry-package/scrivener', 'x-amz-request-id' => '0225463F65FD4273', 'x-amz-version-id' => '_nXYilAqi5bQthzoazNrDhLkxgjDWXKR', 'x-cache' => 'HIT, HIT', 'x-cache-hits' => '1, 1', 'x-served-by' => 'cache-iad2148-IAD, cache-mdw17327-MDW', 'x-timer' => 'S1557334631.288654,VS0,VE4'}}}}
State: %{closing_fun: nil, ets: #Reference<0.2815982234.3043360775.228073>, fetched: #MapSet<[{"hexpm", "accept"}, {"hexpm", "bypass"}, {"hexpm", "idna"}, {"hexpm", "parse_trans"}, {"hexpm", "poison"}, {"hexpm", "ranch"}, {"hexpm", "uri_query"}]>, path: "/root/.hex/cache.ets", pending: #MapSet<[{"hexpm", "phoenix_ecto"}, {"hexpm", "plug_crypto"}, {"hexpm", "file_system"}, {"hexpm", "artificery"}, {"hexpm", "plug_cowboy"}, {"hexpm", "dialyxir"}, {"hexpm", "db_connection"}, {"hexpm", "mime"}, {"hexpm", "bunt"}, {"hexpm", "decimal"}, {"hexpm", "inflex"}, {"hexpm", "bureaucrat"}, {"hexpm", "faker"}, {"hexpm", "mox"}, {"hexpm", "ex_json_schema"}, {"hexpm", "telemetry"}, {"hexpm", "uuid"}, {"hexpm", "mochiweb"}, {"hexpm", "earmark"}, {"hexpm", "phoenix_live_reload"}, {"hexpm", "prometheus_ex"}, {"hexpm", "credo"}, {"hexpm", "ex_doc"}, {"hexpm", "prometheus_process_collector"}, {"hexpm", "ex_machina"}, {"hexpm", "ssl_verify_fun"}, {"hexpm", "unicode_util_compat"}, {"hexpm", "ecto_sql"}, {"hexpm", "makeup_elixir"}, {"hexpm", "hackney"}, {"hexpm", "connection"}, {"hexpm", "metrics"}, {"hexpm", "guardian"}, {"hexpm", "exactor"}, {"hexpm", "scrivener"}, {"hexpm", "prometheus_plugs"}, {"hexpm", "timex"}, {"hexpm", "httpoison"}, {"hexpm", "prometheus_phoenix"}, {"hexpm", "certifi"}, {"hexpm", "meck"}, {"hexpm", "kafka_ex"}, {"hexpm", "exldap"}, {"hexpm", ...}, {...}, ...]>, waiting: %{{"hexpm", "scrivener"} => [{{#PID<0.92.0>, #Reference<0.2815982234.3043229701.229363>}, #Function<3.96831690/0 in Hex.Registry.Server.handle_call/3>}]}}
** (UndefinedFunctionError) function :data_error.exception/1 is undefined (module :data_error is not available)
    (hex) lib/hex/registry/server.ex:59: Hex.Registry.Server.versions/2
    (hex) lib/hex/remote_converger.ex:424: Hex.RemoteConverger.verify_dep/3
    (elixir) lib/enum.ex:775: anonymous fn/3 in Enum.each/2
    (stdlib) maps.erl:257: :maps.fold_1/3
    (elixir) lib/enum.ex:1956: Enum.each/2
    (hex) lib/hex/remote_converger.ex:40: Hex.RemoteConverger.converge/2
    (mix) lib/mix/dep/converger.ex:95: Mix.Dep.Converger.all/4

My hunch is that the data is getting corrupted somehow? Or maybe there is something wrong with the zip version in the system? But Hex could possibly handle this with a better error message.

Earlier I was getting errors that indicated the public key had been revoked and was getting directed here: https://hex.pm/docs/public_keys

Deleting my cached docker images seemed to resolve that. Any recommendations for further debugging? That’s the entire stack track so I’m not really sure where to start looking.