Erlang dependency doesn't compile with mix

Hi guys,

I have an Erlang dependency which does not compile with mix but compiles fine with rebar3. The output when compiling with DEBUG=1 is:

...
===> Building escript...
===> Creating escript file /Users/iulian/elixir/riak/deps/cuttlefish/_build/default/bin/cuttlefish
===> processing <<"cuttlefish">>
===> new deps of <<"cuttlefish">> found to be []
===> processing <<"getopt">>
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badmatch,error}
===> Stack trace to the error location:
[{rebar_prv_escriptize,find_deps_of_deps,3,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,246}]},
 {rebar_prv_escriptize,find_deps,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,240}]},
 {rebar_prv_escriptize,escriptize,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,104}]},
 {rebar_prv_escriptize,do,1,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,80}]},
 {rebar_core,do,2,
             [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar_hooks,run_provider_hooks_,6,
              [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,49}]},
 {rebar_hooks,run_all_hooks,6,
              [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,16}]},
 {rebar_prv_compile,compile,3,
                    [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_compile.erl"},
                     {line,129}]}]
...

I am running Elixir 1.4.5, Erlang/OTP 20 on macOS Sierra. I also made sure the rebar3 mix uses is the correct version using mix local.rebar rebar3. The rebar3 version is 3.4.2.

Any help would be much appreciated!

Getopt uses legacy rebar, try setting manager rebar for it.

I set {:getopt, "~> 0.8.2", manager: :rebar3, override: true}, however I still get the same error.

I said rebar, not rebar3.

Unfortunately same situation. I also made sure to update mix local.rebar rebar.

$ rebar --version
rebar 2.6.4 19 20161104_120312 No VCS info available.

Again this only happens if I compile with mix. If I compile the dependency with rebar3 it compiles.

EDIT: I now have {:getopt, "~> 0.8.2", manager: :rebar, override: true}

If it helps, the dependency is https://hex.pm/packages/riak_core_ng which in turn depends on https://hex.pm/packages/cuttlefish (the culprit in this discussion).

I had similar, but not the same issue before, also I use linux not OSX
I managed to solve it by reinstalling erlang and elixir.

https://elixirforum.com/t/cannot-compile-unicode-util-compat/6724

Well I also tried that but it didn’t work. It’s very frustrating now. If I compile getopt with mix using mix deps.compile getopt it works. If I compile them individually it works except for cuttlefish which gives that error. This is so strange.

I again tried to compile the whole thing with rebar3 and it works fine. I even compiled with ~/.mix/rebar3 for sanity check to make sure mix is not using some strange version.

Ok - grasping at straws here:

On Digital Ocean/Ubuntu there seems to be an emerging pattern that compiling certain applications with rebar3 through mix can run into trouble in a memory constrained environment (512 MB RAM).

https://github.com/elixir-lang/elixir/issues/3857#issuecomment-220746100

Now, I haven’t seen anything comparable for OS X - but it may be worth firing up Activity Monitor and watching the “Memory” tab to see what is happening to the physical memory consumption during compilation:

  • Is physical memory being completely consumed?
  • Is a swap file being created to cover the short fall?

Interesting. I observed the memory usage. It doesn’t really go up and there is no swapping. It is worth noting I have 16GB of RAM. The usage is constant at about 8GB.

This is also followed in this issue: https://github.com/Kyorai/riak_core/issues/23.

2 Likes

Well, that rule’s that out.

Though I wonder if there is a way to see exactly what information is passing from mix to rebar3 - in case rebar3 is simply choking on some “bad (truncated?) configuration information”.

This is the full output:

$ DEBUG=1 mix compile
===> 20.0 satisfies the requirement for minimum OTP version R16
===> Expanded command sequence to be run: []
===> Expanded command sequence to be run: [{default,app_discovery},
                                           {bare,compile}]
===> 20.0 satisfies the requirement for minimum OTP version R16
===> Compiling cuttlefish
===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, compile) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, erlc_compile) -> no hooks defined

===> erlopts [debug_info,warnings_as_errors,
              {parse_transform,lager_transform},
              debug_info,warn_untyped_record]
===> files to compile ["/Users/iulian/elixir/riak/deps/cuttlefish/src/lager_stderr_backend.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_vmargs.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_variable.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_validator.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_util.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_unit.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_translation.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_schema.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_rebar_plugin.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_mapping.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_generator.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_flag.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_escript.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_error.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_enum.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_effective.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_duration_parse.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_duration.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_datatypes.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_conf.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_bytesize.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish_advanced.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/cuttlefish.erl",
                       "/Users/iulian/elixir/riak/deps/cuttlefish/src/conf_parse.erl"]
===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", post_hooks, erlc_compile) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, app_compile) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", post_hooks, app_compile) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, escriptize) -> no hooks defined

===> run_hooks("/Users/iulian/elixir/riak/deps/cuttlefish", pre_hooks, escriptize) -> no hooks defined

===> Building escript...
===> Creating escript file /Users/iulian/elixir/riak/deps/cuttlefish/_build/default/bin/cuttlefish
===> processing <<"cuttlefish">>
===> new deps of <<"cuttlefish">> found to be []
===> processing <<"getopt">>
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badmatch,error}
===> Stack trace to the error location:
[{rebar_prv_escriptize,find_deps_of_deps,3,
                       [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,246}]},
 {rebar_prv_escriptize,find_deps,2,
                       [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,240}]},
 {rebar_prv_escriptize,escriptize,2,
                       [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,104}]},
 {rebar_prv_escriptize,do,1,
                       [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,80}]},
 {rebar_core,do,2,
             [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar_hooks,run_provider_hooks_,6,
              [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,49}]},
 {rebar_hooks,run_all_hooks,6,
              [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,16}]},
 {rebar_prv_compile,compile,3,
                    [{file,"/private/tmp/rebar@3-20170715-74378-1pwzyot/rebar3-3.4.2/_build/default/lib/rebar/src/rebar_prv_compile.erl"},
                     {line,129}]}]
===> When submitting a bug report, please include the output of `rebar3 report "your command"`
** (Mix) Could not compile dependency :cuttlefish, "/Users/iulian/.mix/rebar3 bare compile --paths "/Users/iulian/elixir/riak/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile cuttlefish", update it with "mix deps.update cuttlefish" or clean it with "mixdeps.clean cuttlefish"
1 Like

If I look in the rebar.config of cuttlefish there are some provider hooks: {provider_hooks, [{post, [{compile, {default, escriptize}}]}]}.. If I comment out this line it compiles with mix. Seems that escriptize is looking for some files in the wrong locations or something…

I would guess this is a bug with mix, unless I have something set up incorrectly.

FYI: This rebar3 issue seems to have a similar failure pattern.