Error “Failed to load NIF library” in CI when using Rustler with dynamic libraries

We have a deps section in our CI yaml that looks something like what’s below. It looks like you are missing the rust tool chain, cargo deps fetch and rust compilation steps.

  deps:
    name: Dependencies
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          submodules: true

      - name: Setup Elixir
        uses: erlef/setup-beam@v1
        with:
          version-file: .tool-versions
          version-type: strict

      - name: Setup Rust
        uses: dtolnay/rust-toolchain@stable
        with:
          toolchain: stable

      - name: Retrieve Mix dependencies
        uses: actions/cache@v4
        id: cache-compiled-deps
        with:
          path: |
            deps
            _build
          key: ${{ runner.os }}-${{ env.MIX_ENV }}-${{ hashFiles('mix.lock') }}-v2

      - name: Retrieve Cargo dependencies
        uses: Swatinem/rust-cache@v2
        with:
          cache-on-failure: true
          shared-key: rustler-v2
          workspaces: native/imagesize -> target

      - name: Install Mix dependencies
        run: mix deps.get

      - name: Compile Mix dependencies.
        run: mix deps.compile

      - name: Compile Rust dependencies
        working-directory: native/imagesize
        run: cargo build --profile release

      - name: Create Dialyzer PLT files
        run: mix dialyzer --plt
        if: ${{ false }}

Hey there.

I think your setup would work with a simpler rust library because looking at your CI output you can see that rust dependencies are downloaded and rust is also compiling stuff.

The extractous rust library is doing (compiling) stuff with GraalVM because of Apache Tika, see extractous/extractous-core/README.md at main · yobix-ai/extractous · GitHub

I also don’t get it running locally, even with a newer rustler version with the deprecations fixed and a GraalVM Java SDK available.

It compiles without errors but also fails with the same Failed to load NIF library error.

Maybe someone has an idea what happens with the libtika_native.so file and how to handle it with a rustler setup.

@codestirring I think you are the article author? Can you tell us how you were able to get it running?

Thx

Cheers
Frank

Appreciate the response! Oddly still failure to load with mix compile

ci.yaml
failure to load

Great point. This might be specific to this rust package. I can’t figure this out either in CI. I can run it on my Mac but I haven’t discerned why that is.

Ah, ok I only tried on linux.

But I just build a simple rust CLI with extractous usage like in the article which works without complaining about libtika_native.so like with rustler.

So there might be some steps needed when used with rustler I can’t figure our right now.

https://github.com/snewcomer/nif-extractous

Yes I think you are right about rustler. Building it in Github Action is trying to reference that libtika_native.so from that /runner/ folder. At this point, I am at a loss.

[warning] The on_load function for module Elixir.NifExtractous.Native returned:
{:error,
 {:load_failed,
  ~c"Failed to load NIF library: 'dlopen(/Users/snewcomer/github/my_site/_build/dev/lib/nif_extractous/priv/native/libnif_extractous-v0.3.2-nif-2.17-aarch64-apple-darwin.so, 0x0002): Library not loaded: /Users/runner/work/nif-extractous/nif-extractous/target/aarch64-apple-darwin/release/build/extractous-1392444ace32bce9/out/tika-native/build/native/nativeCompile/libtika_native.dylib\n  Referenced from: <D02FBE3D-DA3C-3332-9BD2-76946358E9A3> /Users/snewcomer/github/my_site/_build/dev/lib/nif_extractous/priv/native/libnif_extractous-v0.3.2-nif-2.17-aarch64-apple-darwin.so\n  Reason: tried: '/Users/runner/work/nif-extractous/nif-extractous/target/aarch64-apple-darwin/release/build/extractous-1392444ace32bce9/out/tika-native/build/native/nativeCompile/libtika_native.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/work/nif-extractous/nif-extractous/target/aarch64-apple-darwin/release/build/extractous-1392444ace32bce9/out/tika-native/build/native/nativeCompile/libtika_native.dylib' (no such file), '/Users/runner/work/nif-extractous/nif-extractous/target/aarch64-apple-darwin/release/build/extractous-1392444ace32bce9/out/tika-native/build/native/nativeCompile/libtika_native.dylib' (no such file)'"}

Rustler is not aware of the additional libraries that are copied in extractous’s build.rs. You will have to extend your build process to copy them from Rust’s target directory to priv/native/....

Thanks @filmor! Anybody aware of a precedent in the rustler community that we could pull from for this extractous rust library?

A little more investigation…

 ldd _build/prod/rel/**/priv/native/libdoc_extractor.so

	linux-vdso.so.1 (0x0000ffff82483000)
	libtika_native.so => not found
	libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff82380000)
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff821d0000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff8244a000)

So it is just that one libtika_native.so binary. Nothing else. Extending the build process seems like a good path here.

https://github.com/snewcomer/nif-extractous

Alright this ended up working. See README for what I had to do!