Erlang dependency doesn't compile with mix

Iulian7 · July 20, 2017, 5:20pm

Hi guys,

I have an Erlang dependency which does not compile with mix but compiles fine with rebar3. The output when compiling with DEBUG=1 is:

...
===> Building escript...
===> Creating escript file /Users/iulian/elixir/riak/deps/cuttlefish/_build/default/bin/cuttlefish
===> processing <<"cuttlefish">>
===> new deps of <<"cuttlefish">> found to be []
===> processing <<"getopt">>
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badmatch,error}
===> Stack trace to the error location:
[{rebar_prv_escriptize,find_deps_of_deps,3,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,246}]},
 {rebar_prv_escriptize,find_deps,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,240}]},
 {rebar_prv_escriptize,escriptize,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,104}]},
 {rebar_prv_escriptize,do,1,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,80}]},
 {rebar_core,do,2,
             [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar_hooks,run_provider_hooks_,6,
              [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,49}]},
 {rebar_hooks,run_all_hooks,6,
              [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,16}]},
 {rebar_prv_compile,compile,3,
                    [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_compile.erl"},
                     {line,129}]}]
...

I am running Elixir 1.4.5, Erlang/OTP 20 on macOS Sierra. I also made sure the rebar3 mix uses is the correct version using mix local.rebar rebar3. The rebar3 version is 3.4.2.

Any help would be much appreciated!

NobbZ · July 20, 2017, 5:41pm

Getopt uses legacy rebar, try setting manager rebar for it.

Iulian7 · July 20, 2017, 6:37pm

I set {:getopt, "~> 0.8.2", manager: :rebar3, override: true}, however I still get the same error.

NobbZ · July 20, 2017, 6:43pm

I said rebar, not rebar3.

Iulian7 · July 20, 2017, 7:30pm

Unfortunately same situation. I also made sure to update mix local.rebar rebar.

$ rebar --version
rebar 2.6.4 19 20161104_120312 No VCS info available.

Again this only happens if I compile with mix. If I compile the dependency with rebar3 it compiles.

EDIT: I now have {:getopt, "~> 0.8.2", manager: :rebar, override: true}

Iulian7 · July 20, 2017, 7:58pm

If it helps, the dependency is https://hex.pm/packages/riak_core_ng which in turn depends on https://hex.pm/packages/cuttlefish (the culprit in this discussion).

georosz · July 21, 2017, 5:39am

I had similar, but not the same issue before, also I use linux not OSX
I managed to solve it by reinstalling erlang and elixir.

https://elixirforum.com/t/cannot-compile-unicode-util-compat/6724

Iulian7 · July 21, 2017, 1:35pm

Well I also tried that but it didn’t work. It’s very frustrating now. If I compile getopt with mix using mix deps.compile getopt it works. If I compile them individually it works except for cuttlefish which gives that error. This is so strange.

I again tried to compile the whole thing with rebar3 and it works fine. I even compiled with ~/.mix/rebar3 for sanity check to make sure mix is not using some strange version.

peerreynders · July 21, 2017, 2:09pm

Ok - grasping at straws here:

On Digital Ocean/Ubuntu there seems to be an emerging pattern that compiling certain applications with rebar3 through mix can run into trouble in a memory constrained environment (512 MB RAM).

github.com/elixir-lang/elixir

(Mix) Could not compile dependency idna, /root/.mix/rebar command failed. If you want to recompile this dependency, please run: mix deps.compile idna

opened 05:06PM - 10 Oct 15 UTC

closed 10:26PM - 10 Oct 15 UTC

edossantos

mix.exs file defmodule Wham.Mixfile do use Mix.Project def project do … [app: :wham, version: "0.0.1", elixir: "~> 1.0", elixirc_paths: elixirc_paths(Mix.env), compilers: [:phoenix] ++ Mix.compilers, build_embedded: Mix.env == :prod, start_permanent: Mix.env == :prod, deps: deps] end # Configuration for the OTP application # # Type `mix help compile.app` for more information def application do [mod: {Wham, []}, applications: [:phoenix, :phoenix_html, :cowboy, :logger, :phoenix_ecto, :postgrex, :maru, :maru_swagger, :tzdata]] end # Specifies which paths to compile per environment defp elixirc_paths(:test), do: ["lib", "web", "test/support"] defp elixirc_paths(_), do: ["lib", "web"] # Specifies your project dependencies # # Type `mix help deps` for examples and options defp deps do [{:phoenix, "~> 1.0.2"}, {:phoenix_ecto, "~> 1.1"}, {:postgrex, ">= 0.0.0"}, {:phoenix_html, "~> 2.1"}, {:phoenix_live_reload, "~> 1.0", only: :dev}, {:cowboy, "~> 1.0"}, {:maru, "~> 0.8.1"}, {:maru_swagger, "~> 0.5", only: :dev }, {:comeonin, "~> 1.0"}, {:timex, "~> 1.0.0-pre"}] end end

Now, I haven’t seen anything comparable for OS X - but it may be worth firing up Activity Monitor and watching the “Memory” tab to see what is happening to the physical memory consumption during compilation:

Is physical memory being completely consumed?
Is a swap file being created to cover the short fall?

Iulian7 · July 21, 2017, 2:31pm

Interesting. I observed the memory usage. It doesn’t really go up and there is no swapping. It is worth noting I have 16GB of RAM. The usage is constant at about 8GB.

This is also followed in this issue: https://github.com/Kyorai/riak_core/issues/23.

peerreynders · July 21, 2017, 2:42pm

Well, that rule’s that out.

Though I wonder if there is a way to see exactly what information is passing from mix to rebar3 - in case rebar3 is simply choking on some “bad (truncated?) configuration information”.

Iulian7 · July 21, 2017, 2:57pm

This is the full output:

$ DEBUG=1 mix compile
===> 20.0 satisfies the requirement for minimum OTP version R16
===> Expanded command sequence to be run: []
===> Expanded command sequence to be run: [{default,app_discovery},
                                           {bare,compile}]
===> 20.0 satisfies the requirement for minimum OTP version R16
===> Compiling cuttlefish
===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, compile) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, erlc_compile) -> no hooks defined

===> erlopts [debug_info,warnings_as_errors,
              {parse_transform,lager_transform},
              debug_info,warn_untyped_record]
===> files to compile ["/Users/iulian/elixir/riak/deps/cuttlefish/src/lager_stderr_backend.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_vmargs.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_variable.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_validator.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_util.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_unit.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_translation.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_schema.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_rebar_plugin.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_mapping.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_generator.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_flag.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_escript.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_error.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_enum.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_effective.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_duration_parse.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_duration.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_datatypes.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_conf.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_bytesize.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_advanced.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/conf_parse.erl"]
===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", post_hooks, erlc_compile) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, app_compile) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", post_hooks, app_compile) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, escriptize) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, escriptize) -> no hooks defined

===> Building escript...
===> Creating escript file /Users/iulian/elixir/riak/deps/cuttlefish/_build/default/bin/cuttlefish
===> processing <<"cuttlefish">>
===> new deps of <<"cuttlefish">> found to be []
===> processing <<"getopt">>
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badmatch,error}
===> Stack trace to the error location:
[{rebar_prv_escriptize,find_deps_of_deps,3,
                       [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,246}]},
 {rebar_prv_escriptize,find_deps,2,
                       [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,240}]},
 {rebar_prv_escriptize,escriptize,2,
                       [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,104}]},
 {rebar_prv_escriptize,do,1,
                       [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,80}]},
 {rebar_core,do,2,
             [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar_hooks,run_provider_hooks_,6,
              [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,49}]},
 {rebar_hooks,run_all_hooks,6,
              [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,16}]},
 {rebar_prv_compile,compile,3,
                    [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_compile.erl"},
                     {line,129}]}]
===> When submitting a bug report, please include the output of `rebar3 report "your command"`
** (Mix) Could not compile dependency :cuttlefish, "/Users/iulian/.mix/rebar3 bare compile --paths "/Users/iulian/elixir/riak/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile cuttlefish", update it with "mix deps.update cuttlefish" or clean it with "mixdeps.clean cuttlefish"

Iulian7 · July 21, 2017, 3:05pm

If I look in the rebar.config of cuttlefish there are some provider hooks: {provider_hooks, [{post, [{compile, {default, escriptize}}]}]}.. If I comment out this line it compiles with mix. Seems that escriptize is looking for some files in the wrong locations or something…

I would guess this is a bug with mix, unless I have something set up incorrectly.

peerreynders · July 21, 2017, 3:55pm

FYI: This rebar3 issue seems to have a similar failure pattern.