Failed to load NIF library (libstdc++.so.6 needed by libexla.so) in Burrito release

Elixir Framework Forums Nx Forum
, , , ,
1

Hi, I built a small cli application combining Bumblebee and Burrito you can find the repository here: GitHub - joelpaulkoch/bumbur: An example for an AI CLI application in Elixir

You can build binaries using MIX_ENV=prod mix release, they will appear in the burrito_out directory.

You can cross compile but I’m building only the executable for the system I’m working on, so this is not a cross compilation issue.

The binary produced for macos works perfectly but I get this error for the linux executable:

19:17:06.520 [warning] The on_load function for module Elixir.EXLA.NIF returned:
{:error,
 {:load_failed,
  ~c"Failed to load NIF library: 'Error loading shared library libstdc++.so.6: No such file or directory (needed by /home/joel/.local/share/.burrito/bumbur_erts-15.0.1_0.1.0/lib/exla-0.7.3/priv/libexla.so)'"}}

Running ldd /home/joel/.local/share/.burrito/bumbur_erts-15.0.1_0.1.0/lib/exla-0.7.3/priv/libexla.so produces this output:

linux-vdso.so.1 (0x00007fd28a598000)
libxla_extension.so => /home/joel/.local/share/.burrito/bumbur_erts-15.0.1_0.1.0/lib/exla-0.7.3/priv/xla_extension/lib/libxla_extension.so (0x00007fd279e00000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fd279a00000)
libm.so.6 => /lib64/libm.so.6 (0x00007fd279d17000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd279ce9000)
libc.so.6 => /lib64/libc.so.6 (0x00007fd279600000)
librt.so.1 => /lib64/librt.so.1 (0x00007fd28a422000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd28a41d000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fd28a418000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd28a59a000)

libstdc++.so.6 is on my system at /lib64.
So I tried running LD_LIBRARY_PATH=/lib64 burrito_out/bumbur_linux, this results in another error:

19:21:20.264 [warning] The on_load function for module Elixir.EXLA.NIF returned:
{:error,
 {:load_failed,
  ~c"Failed to load NIF library: 'Error relocating /home/joel/.local/share/.burrito/bumbur_erts-15.0.1_0.1.0/lib/exla-0.7.3/priv/libexla.so: __libc_single_threaded: symbol not found'"}}

I’m not sure what the underlying issue is.
I believe that _build/prod/lib/exla/priv/libexla.so is copied into the Burrito build and that somehow doesn’t play nicely.
I’m not even sure if this is actually an issue of Burrito or Exla, or if I’m simply missing a flag.

What I’ve tried:

  • run LD_LIBRARY_PATH=/lib64 burrito_out/bumbur_linux
  • set nifcxx_flags: "-L/lib64" for the Burrito target (same for nifc_flags)

Burrito uses musl, I don’t know if this plays a role here.

The situation looks similar to this post which unfortunately doesn’t include a solution: Using EXLA in mix release on MacOS/Darwin fails with `Failed to load NIF library: ... libxla_extension.so...`

Any help is welcome.

2

Can you confirm this is happening when cross-compiling both from mac/linux?

3

This is probably because Burrito uses musl. The libxla_extension.so that libexla.so uses precompiled on Linux is only precompiled against glibc. There was an Alpine/musl build a long time ago, but it was way too hard to maintain because it relied on patching TensorFlow/XLA and LLVM at the same time: Add alpine build by seanmor5 · Pull Request #31 · elixir-nx/xla · GitHub

1 Like
4

I can cross compile successfully for both platforms on both systems.
I get this error at runtime on my linux system with the binary built on the linux system.
I’ve disabled cross compiling on the linux system, removed all caches and only built the linux binary, it’s the same result.
I didn’t try to run the linux binary built on macos.

5

This is my current understanding:

  • I get the precompiled libexla.so which is precompiled against glibc
  • I can run the program using mix run, this way /lib64/libstdc++.so.6 should be used (?)
  • When I build the burrito release, libexla.so is packed into the binary together with everything else
  • When I run the burrito release for the first time, everything gets unpacked into ~/.local/share/.burrito/, moreover burrito copies a runtime library to /tmp/libc-musl-[some hash].so
  • libexla.so fails to find libstdc++.so.6, it’s unchanged so /lib64 is not passed as a directory to look for shared libraries
  • When I add LD_LIBRARY_PATH=/lib64, libstdc++.so.6 is found but I still get the error __libc_single_threaded: symbol not found because now I’m running against system libraries at /lib64 and the burrito musl runtime library at /tmp at the same time

I guess possible solutions are:

  1. link everything exla related against the burrito musl runtime library
  2. build a statically linked libexla.so

Not sure if I will put any more work into this but thanks to both of you nevertheless.

6

Quick question, what architecture are you compiling against? New macs are running on ARM while your usual linux device is x86_64.

7

I compile on a x86_64 linux machine for a x86_64 linux machine, so I don’t think it’s related to cross compiling.

1 Like
8

I had a similar issue when attempting to use an older version of Linux (RHEL 7) to build a release.

Are all your Linux deps up to date? And is your OS at EOL or is it still getting updates?

Just a shot in the dark…

9

Thanks, good idea, but it’s a rolling distro, so everything is up to date