How to make an Elixir library available to erlang users?

PragTob · May 21, 2017, 2:04pm

Hello Elixir friends,

I want to make one of my Elixir available/usable for our Erlang friends and be a “good BEAM citizen”. I was also told, that there isn’t really a good benchmarking library for Erlang so it seemed like a great idea.

Problem: I can’t figure out how to install Elixir libraries in Erlang. Threads like Writing a library for use in both Elixir and Erlang as well as the prominent Erlang/rebar3 usage on hex.pm made me think this was possible (although I always sort of wondered where it gets all the Elixir from)…

I setup a little test respository. However, benchee can not be found and neither can any other popular elixir hex package I can think of:

tobi@speedy ~/github/benchee_erlang $ rebar3 update
===> Updating package registry...
===> Writing registry to /home/tobi/.cache/rebar3/hex/default/registry
===> Generating package index...
===> Writing index to /home/tobi/.cache/rebar3/hex/default/packages.idx
tobi@speedy ~/github/benchee_erlang $ rebar3 compile
===> Package <<"benchee">> not found. Fetching registry updates and trying again...
===> Updating package registry...
===> Writing registry to /home/tobi/.cache/rebar3/hex/default/registry
===> Generating package index...
===> Writing index to /home/tobi/.cache/rebar3/hex/default/packages.idx
===> Package not found in registry: <<"benchee">>.
tobi@speedy ~/github/benchee_erlang $ rebar3 pkgs | grep poison
tobi@speedy ~/github/benchee_erlang $ rebar3 pkgs | grep phoenix

(btw. yes I have the rebar3_hex plugin installed but I think it does something else)

Adding benchee as a git dependency I get this error:

tobi@speedy ~/github/benchee_erlang $ rebar3 compile
===> Verifying dependencies...
===> Fetching benchee ({git,"git://github.com/PragTob/benchee.git"})
===> WARNING: It is recommended to use {branch, Name}, {tag, Tag} or {ref, Ref}, otherwise updating the dep may not work as expected.
===> Dependency failure: Application benchee not found at the top level of directory /home/tobi/github/benchee_erlang/_build/default/lib/benchee

It’s from this rebar3 issue - basically I don’t follow the standard rebar3/erlang project structure and yeah I’m an elixir project, so sure it doesn’t work.

So what do I do? Elixir and Erlang/rebar3 both use hex.pm and elixir people can use Erlang stuff but not the other way around? Do I just not know how?

How can I allow an Erlang project to use an Elixir library? Do they have to copy the files and compile them themselves to beam files? Do I setup a separate rebar3 package?

Sorry my erlang knowledge is also rather… improvable

I feel like (and hope) I’m really missing something so please feel free to point it out. Thank you in advance

idi527 · May 21, 2017, 2:08pm

A bit off-topic, but some erlang programmers would want to avoid using any elixir libs because elixir needs to include it’s standard library at runtime, so if someone wants their library to be used by all languages on beam, it might be better to write it in erlang. In your case though it probably doesn’t matter, since it’s a benchmarking library.

PragTob · May 21, 2017, 2:11pm

Should they avoid it for that reason, then their loss It’s also not used at run time, so there is no impact on the runtime release that gets distributed

The library has a non trivial code base by now so I’m not gonna rewrite, just trying to be a good BEAM citizen.

josevalim · May 21, 2017, 2:36pm

I believe you need something like rebar3_mix or rebar3_elixir for it to work. Try pinging t_slought/tristan or alisdair on #elixir-lang IRC channel.

PragTob · May 21, 2017, 3:11pm

Thanks, will do/try!

PragTob · May 21, 2017, 3:31pm

Update: Thanks to @josevalim ( ) I found rebar3_elixir_compile which seems to do what I want it to with the following configuration:

{deps, [
  {benchee, {elixir, "benchee", "0.8.0"}}
  %% {benchee, {git, "git://github.com/PragTob/benchee.git"}}
]}.

{plugins, [
    { rebar3_elixir_compile, ".*", {git, "https://github.com/barrel-db/rebar3_elixir_compile.git", {branch, "master"}}}
]}.

{provider_hooks, [
  {pre, [{compile, {ex, compile}}]},
  {pre, [{release, {ex, compile}}]}
]}.

{elixir_opts,
  [
    {env, dev}
  ]
}.

It fetches all dev dependencies atm and values other than the dev environment don’t seem to work. Will ask there. Seems to work now:

tobi@speedy ~/github/benchee_erlang $ rebar3 shell
===> Verifying dependencies...
===> Upgrading benchee ({elixir,"benchee","0.8.0"})
Dependency benchee already exists

Running dependency resolution...
All dependencies up to date
===> App bunt is no longer needed and can be deleted.
===> App credo is no longer needed and can be deleted.
===> App deep_merge is no longer needed and can be deleted.
===> App dialyxir is no longer needed and can be deleted.
===> App earmark is no longer needed and can be deleted.
===> App ex_doc is no longer needed and can be deleted.
===> App fs is no longer needed and can be deleted.
===> App mix_test_watch is no longer needed and can be deleted.
===> Compiling benchee_erlang
===> Adding Elixir Libs
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V8.1  (abort with ^G)
1> 'Elixir.Benchee':init().
#{'__struct__' => 'Elixir.Benchee.Suite',
  configuration => #{'__struct__' => 'Elixir.Benchee.Configuration',
    assigns => #{},
    formatter_options => #{console => #{comparison => true,unit_scaling => best}},
    formatters => [#Fun<Elixir.Benchee.Formatters.Console.output.1>],
    inputs => nil,
    parallel => 1,
    print => #{benchmarking => true,configuration => true,fast_warning => true},
    time => 5000000,
    warmup => 2000000},
  jobs => #{},
  run_times => nil,
  statistics => nil,
  system => nil}

Thanks a lot!

If someone has other ideas/ways please still let me know

Crowdhailer · May 20, 2018, 1:25pm

It might be time for a new thread on this but I am wonder what the resolution to this thread was. What is the best way to

a) use Elixir libs from an erlang project
b) structure an Elixir project so it is easy to use from erlang

PragTob · May 21, 2018, 7:44am

I think rebar3_elixir_compile is still the goto solution. Albeit no one ever responded to the issues I created but there have been some updates.

My test repo works some of the time - albeit I also needed to escriptize and I’m not too familiar with rebar3/escriptize so others might to a better job than me.

From my perspective for use cases such as benchee (i.e. you don’t need something at runtime but rather just a tool) it’s easiest/most reliable to just install elixir use the library and then call to erlang code from there. I’d be happy for a better solution though.

At Lambdadays I talked quickly with Michał and José and it sounded like it’d be nice to do something to make it easier and that it shouldn’t be that hard but also not a top priority right now.

Crowdhailer · May 21, 2018, 7:04pm

Thanks for the test repo. I hope it is something that can be prioritised soon. I would really like to see the beam community as people like to call it become more cohesive

josevalim · May 23, 2018, 4:09pm

Thanks @PragTob! I want to clarify that people should not expect the Elixir team to solve this. We already do all of the work to ensure you can use rebar/rebar3/make from Mix, the other direction is not quite up to us but rather to people using those other build tools. We will be glad to answer any questions or improve Elixir if necessary but it is up to the community to tackle the problem together.

tristan · April 27, 2019, 1:22pm

rebar_mix https://github.com/tsloughter/rebar_mix is the newer solution that handles transitive deps.

tristan · April 27, 2019, 1:24pm

Oh, this thread is a year old, haha. Someone had just linked me to it.

PragTob · April 28, 2019, 6:11pm

Still good news and worth adding here if people find it through Google or what not. Thanks for chiming in I might try again with benchee

kip · July 8, 2020, 8:26am

Building on this thread I’d like to make some of my CLDR libs more easily consumable by erlang and lfe. As I read it the rebar3 plugin rebar_mix should take care of the compile phase for erlang. But the module naming is a little bit jarring on the other side. I am very comfortable using erlang modules in Elixir-land but I’m not sure that the reverse is true for erlangers…

For the erlangers here, what would you advise? Shim the Elixir modules with erlang style module names? No change and just suggest people get over it? Any other ideas?

ntalfer · September 3, 2020, 7:33am

In Erlang, module names must be atoms. For simplicity, to avoid the simple quotes, the common way is to use a module name with lower case characters, underscores and numbers only. The same thing is true for record names.

I understand it might be boring typing 'Elixir.X.Y':some_fun() from erlang source code, especially it there are many calls.
If someone’s not happy with that, there’s a quite simple solution which consists in adding a define for the module name and use it.

-define(LIB,  'Elixir.X.Y').

and perform calls like this:

?LIB:some_fun(),
?LIB:some_other_fun(),

toraritte · November 13, 2021, 5:29pm

Apologies for the necrobump, but came across the Adopting Erlang online book, and there is a specific section in there about this topic (Development > Dependencies > Using Elixir Dependencies) that walks the reader through how to set up rebar_mix in Rebar3 projects to pull in Elixir hex.pm packages into Erlang applications.