How to install erlang NIF for use in Livebook?

bt-maps · January 27, 2023, 12:41pm

Hi,

I am new to the Elixir/Erlang ecosystem. I found an Erlang NIF here that I would like to use as is in an interactive Livebook and am unsure how to proceed.

Is it possible to install this directly from Mix.install in Livebook, or do I need a more involved process with Mix/Rebar?

Thanks!
BT

kwando · January 27, 2023, 12:47pm

I do not know the answer, but have you tried to use Mix.install([:h3]) in your Livebook?

D4no0 · January 27, 2023, 12:49pm

NIF is core functionality of erlang, you can use it out of the box without any dependencies, you can read the guide.

I’m not sure how you intend to use it, however in livebook usecase you need to have everything precompiled, as I don’t think you will be able to compile c source files from livebook.

kwando · January 27, 2023, 12:52pm

This is running in a Livebook, to me it looks like it is compiling some things

Mix.install([:h3])

Output:

Resolving Hex dependencies...
Dependency resolution completed:
New:
  h3 3.6.4
* Getting h3 (Hex package)
-- The C compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Setting build type to 'Release' as none was specified.
-- Found Erlang: /usr/local/lib/erlang/erts-12.3.2.2/bin/erl  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src
Scanning dependencies of target external-h3
[ 10%] Creating directories for 'external-h3'
[ 20%] Performing download step (git clone) for 'external-h3'
Cloning into 'external-h3'...
warning: redirecting to https://github.com/uber/h3.git/
Note: switching to 'v3.6.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 8e745ce3 Merge pull request #307 from nrabinowitz/release-3.6.3
[ 30%] Performing update step for 'external-h3'
[ 40%] No patch step for 'external-h3'
[ 50%] Performing configure step for 'external-h3'
-- The C compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Warning at CMakeLists.txt:249 (message):
  clang-format was not detected, so automatic source code reformatting is
  disabled


CMake Warning at CMakeLists.txt:261 (message):
  clang-tidy was not detected, so source code linting is disabled


-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src/external-h3/src/external-h3-build
[ 60%] Performing build step for 'external-h3'
Scanning dependencies of target h3
[  6%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/algos.c.o
[ 12%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/coordijk.c.o
[ 18%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/bbox.c.o
[ 25%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/polygon.c.o
[ 31%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/h3Index.c.o
[ 37%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/vec2d.c.o
[ 43%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/vec3d.c.o
[ 50%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/linkedGeo.c.o
[ 56%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/localij.c.o
[ 62%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/geoCoord.c.o
[ 68%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/h3UniEdge.c.o
[ 75%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/mathExtensions.c.o
[ 81%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/vertexGraph.c.o
[ 87%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/faceijk.c.o
[ 93%] Building C object CMakeFiles/h3.dir/src/h3lib/lib/baseCells.c.o
[100%] Linking C static library lib/libh3.a
[100%] Built target h3
[ 70%] Performing install step for 'external-h3'
[100%] Built target h3
Install the project...
-- Install configuration: ""
-- Installing: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src/lib/libh3.a
-- Installing: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src/include/h3/h3api.h
-- Installing: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src/lib/cmake/h3/h3Config.cmake
-- Installing: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src/lib/cmake/h3/h3ConfigVersion.cmake
-- Installing: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src/lib/cmake/h3/h3Targets.cmake
-- Installing: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src/lib/cmake/h3/h3Targets-noconfig.cmake
[ 80%] Completed 'external-h3'
[ 80%] Built target external-h3
Scanning dependencies of target h3
[ 90%] Building C object CMakeFiles/h3.dir/h3.c.o
[100%] Linking C shared module libh3.so
[100%] Built target h3
Install the project...
-- Install configuration: "Release"
-- Installing: /home/livebook/.cache/mix/installs/elixir-1.14.0-erts-12.3.2.2/fe0842890a0fbfa8020587c61dfacd08/deps/h3/c_src/../priv/libh3.so
===> Analyzing applications...
===> Compiling h3

bt-maps · January 27, 2023, 1:20pm

Thanks for the response. My install doesn’t make it that far. It’s not finding my cmake install. Do I need rebar3 installed?

sh: line 0: exec: cmake: not found
===> Hook for compile failed!

** (Mix.Error) Could not compile dependency :h3, "/Applications/Livebook.app/Contents/Resources/rel/vendor/rebar3 bare compile --paths /Users/*user*/Library/Caches/mix/installs/elixir-1.14.2-erts-13.0.4/480742e86c2d2b4ef6294c9ea261faf5/_build/dev/lib/*/ebin" command failed. Errors may have been logged above. You may run Mix.install/2 to try again or change the arguments to Mix.install/2 to try another version
    (mix 1.14.2) lib/mix.ex:513: Mix.raise/2
    (elixir 1.14.2) lib/file.ex:1607: File.cd!/2
    (mix 1.14.2) lib/mix/tasks/deps.compile.ex:310: Mix.Tasks.Deps.Compile.do_command/5
    (mix 1.14.2) lib/mix/tasks/deps.compile.ex:219: Mix.Tasks.Deps.Compile.do_rebar3/2
    (mix 1.14.2) lib/mix/tasks/deps.compile.ex:96: anonymous fn/4 in Mix.Tasks.Deps.Compile.compile/2
    (elixir 1.14.2) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
    (elixir 1.14.2) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
    #cell:setup:1: (file)

kwando · January 27, 2023, 1:41pm

You are running the Livebook.app right?

Do you have cmake installed? I got the same error first and it worked after installing cmake. (I used brew install cmake)

bt-maps · January 27, 2023, 1:55pm

Yes, I am running Livebook.app on Mac (Livebook v0.8.1, Elixir v1.14.2. cmake is installed (using brew install cmake) and I also did a brew install rebar3 just in case. Same error about not finding cmake.

Brents-MacBook-Pro-2 ~ % cmake
Usage

  cmake [options] <path-to-source>
  cmake [options] <path-to-existing-build>
  cmake [options] -S <path-to-source> -B <path-to-build>

Brents-MacBook-Pro-2 ~ % rebar3
Rebar3 is a tool for working with Erlang projects.

Usage: rebar3 [-h] [-v] [<task>]

  -h, --help     Print this help.
  -v, --version  Show version information.
  <task>         Task to run.

kwando · January 27, 2023, 2:30pm

If you have elixir on your machine, can you see if this works?

❯ iex
Erlang/OTP 25 [erts-13.0.1] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Interactive Elixir (1.14.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Mix.install([:h3])
:ok

outlog · January 27, 2023, 2:36pm

try doing this:

path = System.get_env("PATH")
System.put_env("PATH", path <> ":/opt/homebrew/bin")

Mix.install(
  [{:h3, "~> 3.6"}]
)

bt-maps · January 27, 2023, 2:44pm

that worked from iex, still not successful in livebook though.

Interactive Elixir (1.14.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Mix.install([:h3])
Resolving Hex dependencies...
Dependency resolution completed:
New:
  h3 3.6.4
* Getting h3 (Hex package)
Could not find "rebar3", which is needed to build dependency :h3
I can install a local copy which is just used by Mix
Shall I install rebar3? (if running non-interactively, use "mix local.rebar --force") [Yn] Y
* creating /Users/user/.mix/elixir/1-14/rebar3
-- The C compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Setting build type to 'Release' as none was specified.
-- Found Erlang: /usr/local/Cellar/erlang/25.2/lib/erlang/erts-13.1.3/bin/erl  
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/user/Library/Caches/mix/installs/elixir-1.14.3-erts-13.1.3/fe0842890a0fbfa8020587c61dfacd08/deps/h3/_build/c_src
[ 10%] Creating directories for 'external-h3'
[ 20%] Performing download step (git clone) for 'external-h3'
Cloning into 'external-h3'...
...
...
...
...
...

[ 80%] Completed 'external-h3'
[ 80%] Built target external-h3
[ 90%] Building C object CMakeFiles/h3.dir/h3.c.o
[100%] Linking C shared module libh3.so
[100%] Built target h3
Install the project...
-- Install configuration: "Release"
-- Installing: /Users/user/Library/Caches/mix/installs/elixir-1.14.3-erts-13.1.3/fe0842890a0fbfa8020587c61dfacd08/deps/h3/c_src/../priv/libh3.so
===> Analyzing applications...
===> Compiling h3
:ok

josevalim · January 27, 2023, 4:09pm

Go to the settings page and add the homebrew bin directory to your PATH variable under the system environment settings. Now Mix.install should be able to find cmake.

bt-maps · January 27, 2023, 4:34pm

Success! Thank you. Much Appreciated.

bt-maps · January 27, 2023, 10:25pm

OK, now that the NIF is installed, I am having trouble with the basics of calling the erlang function from Elixir. I read that to call an Erlang function from Elixir, you call it using the module name as atom.

Errors from my Livebook and the example tests from the erlang NIF are shown below.

Mix.install([
  {:maplibre, "~> 0.1.3"},
  {:kino_maplibre, "~> 0.1.7"},
  {:h3, "~> 3.6"}
])

H3 = :h3.from_string("845ad1bffffffff")
# from erlang test
# validity_test(_Config) ->
     # H3 = h3:from_string("845ad1bffffffff"),
     # 596072861467148287 = H3,
     # true = h3:is_valid(H3),
    # ok.

** (ArgumentError) argument error
    (h3 3.6.4) :h3.from_string("845ad1bffffffff")

Res = :h3.from_geo({40.689167, -74.044444}, 5)
# from erlang tests
# geo_to_h3_test(_Config) ->
    # Res = h3:from_geo({40.689167, -74.044444}, 5),
    # "852a1073fffffff" = h3:to_string(Res),
    # ok.

** (MatchError) no match of right hand side value: 599718752904282111
    (stdlib 4.0.1) erl_eval.erl:496: :erl_eval.expr/6
 : (file)

kokolegorille · January 27, 2023, 11:33pm

Not sure about that, I did not try h3… but when using erlang’s lib, I would use charlist instead of String.

H3 = :h3.from_string("845ad1bffffffff")

maybe try this…

h3 = :h3.from_string('845ad1bffffffff')

Supamic · June 22, 2023, 8:55am

I deployed a Livebook to Fly.io using the link provided on the website and when I try to include :yamerl to my setup dependencies I get a similar error…

Mix.install([
  {:yamerl, "~> 0.10.0"}
])
===> Analyzing applications...
===> Compiling yamerl
Killed
** (Mix.Error) Could not compile dependency :yamerl, "/home/livebook/.mix/elixir/1-14/rebar3 bare compile --paths /home/livebook/.cache/mix/installs/elixir-1.14.2-erts-12.3.2.2/101e287cd48d01d9f9d6dfbb5482eb05/_build/dev/lib/*/ebin" command failed. Errors may have been logged above. You may run Mix.install/2 to try again or change the arguments to Mix.install/2 to try another version
    (mix 1.14.2) lib/mix.ex:513: Mix.raise/2
    (elixir 1.14.2) lib/file.ex:1607: File.cd!/2
    (mix 1.14.2) lib/mix/tasks/deps.compile.ex:310: Mix.Tasks.Deps.Compile.do_command/5
    (mix 1.14.2) lib/mix/tasks/deps.compile.ex:219: Mix.Tasks.Deps.Compile.do_rebar3/2
    (mix 1.14.2) lib/mix/tasks/deps.compile.ex:96: anonymous fn/4 in Mix.Tasks.Deps.Compile.compile/2
    (elixir 1.14.2) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
    (mix 1.14.2) lib/mix/tasks/deps.compile.ex:80: Mix.Tasks.Deps.Compile.compile/2
    (mix 1.14.2) lib/mix/tasks/deps.loadpaths.ex:84: Mix.Tasks.Deps.Loadpaths.deps_check/2

I tried to force reinstall rebar3 in case this was the issue, but no difference.

System.cmd("mix", ["local.rebar --force"])

I determined the distro with…

System.cmd("cat", ["/proc/version"], into: IO.stream())
Linux version 5.12.2 (root@5b3596d5b582) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Mon Oct 3 13:48:56 UTC 2022

I tried installing build-essential & make packages but these were already at the latest versions.

System.cmd("apt-get", ["install","build-essential", "make"], into: IO.stream())
Reading package lists...
Building dependency tree...
Reading state information...
build-essential is already the newest version (12.9).
make is already the newest version (4.3-4.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Any help would be greatly appreciated because I am at a total loss of what configuration to update or dependency to install to get this compilation to work. Also, I’m actually trying to install :crawly (for collaborating and testing in a Livebook sandbox) which uses this as a dependency so switching to an Elixir native YAML parser is not an option.

josevalim · June 22, 2023, 10:00am

I am suspecting you are running out of memory and the process is being killed. You may need a bigger instance.

Supamic · June 22, 2023, 5:49pm

Thank you so much! I was able to run by configuring a swap file

System.cmd("fallocate", ["-l", "4G", "/swapfile"])
System.cmd("chmod", ["600", "/swapfile"])
System.cmd("mkswap", ["/swapfile"])
System.cmd("swapon", ["/swapfile"])
System.cmd("echo", ["'/swapfile none swap sw 0 0'", "|", "tee", "-a", "/etc/fstab"])
Mix.install([
  {:yamerl, "~> 0.10.0"}
])

===> Analyzing applications...
===> Compiling yamerl
:ok