Using Rustler with OTP24

CharlesO · May 19, 2021, 3:35pm

Compiling any app dependent on Rustler is not working on Elixir v12.0 and OTP-24

Anyone having similar issues?

Could not compile :rustler, no "mix.exs", "rebar.config" or "Makefile" (pass :compile as an option to customize compilation, set it to "false" to do nothing)
Unchecked dependencies for environment prod:
* rustler (https://github.com/rusterlium/rustler.git)
  could not find an app file at "c:/Projects/bat/_build/prod/lib/rustler/ebin/rustler.app". This may happen if the dependency was not yet compiled or the dependency indeed has no app file (then you can pass app: false as option)
could not compile dependency :xler, "mix compile" failed. You can recompile this dependency with "mix deps.compile xler", update it with "mix deps.update xler" or clean it with "mix deps.clean xler"

josevalim · May 19, 2021, 5:30pm

Is that v1.12.0 or one of the RCs? We fixed a bug on the RCs but it should be fixed on the final release.

CharlesO · May 19, 2021, 6:32pm

v1.12.0 sir

josevalim · May 19, 2021, 9:21pm

Looking at the Rustler project, it indeed does not have any of mix.exs/rebar.config/makefile at the root, so it makes sense that Elixir does not know how to compile it: GitHub - rusterlium/rustler: Safe Rust bridge for creating Erlang NIF functions

Try using a :sparse checkout on the relevant subdirectory or use the Hex package. I am not sure how it worked before (it shouldn’t have).

dimitarvp · May 19, 2021, 9:45pm

Which version of rustler are you using? Mine is 0.22.0-rc.0. My Elixir+Rust project compiles fine with Erlang 23.3.4 and Elixir 1.12.0-otp-23, but fails to compile with Erlang 24.0 and Elixir 1.12.0-otp-24. The error is:

Caused by:
  process didn't exit successfully: `/Users/dimi/kod/xqlite/_build/dev/rustler_crates/xqlitenif/release/build/rustler-69fb4c768ce25323/build-script-build` (exit code: 101)
  --- stderr
  thread 'main' panicked at 'Erlang version 2.16 not handled, please file a a bug report.', /Users/dimi/.cargo/registry/src/github.com-1ecc6299db9ec823/rustler-0.22.0-rc.0/build.rs:36:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
    (rustler 0.22.0-rc.0) lib/mix/tasks/compile.rustler.ex:89: Mix.Tasks.Compile.Rustler.compile_crate/1
    (elixir 1.11.4) lib/enum.ex:1411: Enum."-map/2-lists^map/1-0-"/2
    (rustler 0.22.0-rc.0) lib/mix/tasks/compile.rustler.ex:28: Mix.Tasks.Compile.Rustler.run/1
    (mix 1.11.4) lib/mix/task.ex:394: Mix.Task.run_task/3
    (mix 1.11.4) lib/mix/tasks/compile.all.ex:90: Mix.Tasks.Compile.All.run_compiler/2
    (mix 1.11.4) lib/mix/tasks/compile.all.ex:70: Mix.Tasks.Compile.All.compile/4
    (mix 1.11.4) lib/mix/tasks/compile.all.ex:57: Mix.Tasks.Compile.All.with_logger_app/2
    (mix 1.11.4) lib/mix/tasks/compile.all.ex:35: Mix.Tasks.Compile.All.run/1

Seems there’s a hard requirement for a maximum Erlang version. I thought of contributing and maybe even taking over rustler in the future (the owners are semi-open to the idea btw) but life and work keep getting in the way.

Still, maybe I can muster a small PR that fixes this problem soon-ish.

TL;DR: It’s not Elixir 1.12.0’s fault. It seems to be rustler insisting on a maximum Erlang version.

CharlesO · May 20, 2021, 1:10am

This has been updated on the main branch but not yet published as a release. if you use the git repo as your dep, you won’t get that error

dimitarvp · May 20, 2021, 8:42am

I am now using Erlang 24.0 and Elixir 1.12.0-otp-24 plus I use rustler directly from its GitHub repo and I am still getting the error. Which versions do you have exactly so it’s working for you?

CharlesO · May 20, 2021, 9:13am

I still have problems, not fully resolved. but at least The main branch of rustler is updated for OTP-24

Could not compile :rustler, no "mix.exs", "rebar.config" or "Makefile" (pass :compile as an option to customize compilation, set it to "false" to do nothing)
==> meeseeks_html5ever
Could not compile :rustler, no "mix.exs", "rebar.config" or "Makefile" (pass :compile as an option to customize compilation, set it to "false" to do nothing)
Unchecked dependencies for environment prod:
* rustler (https://github.com/rusterlium/rustler.git)
  the dependency build is outdated, please run "MIX_ENV=prod mix deps.compile"
could not compile dependency :meeseeks_html5ever, "mix compile" failed. You can recompile this dependency with "mix deps.compile meeseeks_html5ever", update it with "mix deps.update meeseeks_html5ever" or clean it with "mix deps.clean meeseeks_html5ever"

dimitarvp · May 20, 2021, 9:29am

@josevalim or @AstonJ Can you split this mini-thread off of the main one? We kind of got off-topic.

maciejgryka · May 21, 2021, 11:19am

I had the same problem - it all seems to work after you use the git checkout option both in your mix.exs as well as Cargo.toml (a short blog post about this here).

I was missing the Cargo.toml step for a long time leading me to being stuck with 0.22.0-rc.0 there and still getting the version error.

To summarize, I’ve added this to mix.exs (thanks for the sparse tip @josevalim):

{:rustler, git: "git://github.com/rusterlium/rustler.git", branch: "master", sparse: "rustler_mix"},

And this to Cargo.toml:

rustler = {git = "git://github.com/rusterlium/rustler.git", branch="master"}

CharlesO · May 21, 2021, 4:26pm

no luck compiling {:meeseeks, "~> 0.15.1"}, even with the specified updates.

dimitarvp · May 21, 2021, 8:31pm

There’s a 0.22.0-rc.1 release that got out like 45 minutes ago.

@maciejgryka What you have shown have worked for me for Elixir 1.12.0-otp-24 and Erlang 24.0.1. Thanks!

(I still used rustler from the master branch, though.)

CharlesO · May 22, 2021, 6:13pm

Still have the same issues after this update. Previously working code will not compile anymore.


   Compiling quote v1.0.7
   Compiling rustler_codegen v0.22.0-rc.1
error[E0432]: unresolved imports `rustler_sys::TWinDynNifCallbacks`, `rustler_sys::WIN_DYN_NIF_CALLBACKS`
  --> C:\Users\Charles Okwuagwu\.cargo\registry\src\github.com-1ecc6299db9ec823\rustler-0.22.0-rc.1\src\codegen_runtime.rs:16:23
   |
16 | pub use rustler_sys::{TWinDynNifCallbacks, WIN_DYN_NIF_CALLBACKS};
   |                       ^^^^^^^^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^^^^^^ no `WIN_DYN_NIF_CALLBACKS` in the root
   |                       |
   |                       no `TWinDynNifCallbacks` in the root

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
error: could not compile `rustler`

To learn more, run the command again with --verbose.

== Compilation error in file lib/nif.ex ==
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
    (rustler 0.22.0-rc.1) lib/rustler/compiler.ex:36: Rustler.Compiler.compile_crate/2
    lib/nif.ex:2: (module)
    (stdlib 3.15) erl_eval.erl:685: :erl_eval.do_apply/6

dimitarvp · May 22, 2021, 6:26pm

Show us your mix.exs?

CharlesO · May 22, 2021, 8:18pm

defp deps do

    [

      {:tzdata, "~> 1.1.0"},

      {:gen_smtp, "~> 1.0"},

      {:mime, "~> 1.4"},

      {:fs, "~> 4.10.0"},

      # {:circuits_uart, "~> 1.4"},

      {:jason, "~> 1.2"},

      {:meeseeks, "~> 0.15.1"},

      {:decimal, "~> 2.0"},

      {:tesla, "~> 1.3"},

      {:castore, "~> 0.1.0"},

      # {:rustler, git: "git://github.com/rusterlium/rustler.git" branch: "master", sparse: "rustler_mix"},

      {:rustler, "~> 0.22.0-rc.1", override: true}

    ]

  end

dimitarvp · May 22, 2021, 9:12pm

I’d try both things:

Use the commented out line in your mix.exs:

{:rustler, git: "git://github.com/rusterlium/rustler.git" branch: "master", sparse: "rustler_mix"},

Put this into your Cargo.toml as @maciejgryka showed:

rustler = {git = "git://github.com/rusterlium/rustler.git", branch="master"}

myobie · May 27, 2021, 2:03pm

(Gah, I’m sorry to double post, I messed up making an edit.)

You can override the version rustler is looking for with an ENV variable:

$ RUSTLER_NIF_VERSION=2.15 mix compile

This is the easiest way to make progress (I’ve found) without having to re-target any deps or anything.

Am I missing some downsides to doing this? I think rustler works just fine targeting the older nif version in my experience.

dimitarvp · May 27, 2021, 2:05pm

The reason the approach above has been recommended is just one: rustler maintainers seem to be active again and we can expect a stable next release soon.

If not for that factor then I’d go with what you recommended.

myobie · May 27, 2021, 7:46pm

Ah OK, that makes perfect sense and I didn’t realize. Thanks for letting me know