Error “Failed to load NIF library” in CI when using Rustler with dynamic libraries

snewcomer · January 31, 2025, 4:51am

Recently, I came across this wonderful blog post on Parsing PDFs (and more) in Elixir using Rust. However, when trying out Github Actions CI with Rustler, I can’t seem to figure out the right steps to get mix compile to work.

ci.yaml
Failed to load NIF Library error

Warning: 04:44:23.229 [warning] The on_load function for module Elixir.RustReader returned:
{:error,
 {:load_failed,
  ~c"Failed to load NIF library /home/runner/work/elixir_pdf_tutorial/elixir_pdf_tutorial/_build/test/lib/elixir_pdf/priv/native/librustreader: 'libtika_native.so: cannot open shared object file: No such file or directory'"}}

How are people setting up their CI environments with Rustler and Phoenix projects?

mindok · January 31, 2025, 5:43am

We have a deps section in our CI yaml that looks something like what’s below. It looks like you are missing the rust tool chain, cargo deps fetch and rust compilation steps.

  deps:
    name: Dependencies
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          submodules: true

      - name: Setup Elixir
        uses: erlef/setup-beam@v1
        with:
          version-file: .tool-versions
          version-type: strict

      - name: Setup Rust
        uses: dtolnay/rust-toolchain@stable
        with:
          toolchain: stable

      - name: Retrieve Mix dependencies
        uses: actions/cache@v4
        id: cache-compiled-deps
        with:
          path: |
            deps
            _build
          key: ${{ runner.os }}-${{ env.MIX_ENV }}-${{ hashFiles('mix.lock') }}-v2

      - name: Retrieve Cargo dependencies
        uses: Swatinem/rust-cache@v2
        with:
          cache-on-failure: true
          shared-key: rustler-v2
          workspaces: native/imagesize -> target

      - name: Install Mix dependencies
        run: mix deps.get

      - name: Compile Mix dependencies.
        run: mix deps.compile

      - name: Compile Rust dependencies
        working-directory: native/imagesize
        run: cargo build --profile release

      - name: Create Dialyzer PLT files
        run: mix dialyzer --plt
        if: ${{ false }}

suchasurge · January 31, 2025, 2:27pm

Hey there.

I think your setup would work with a simpler rust library because looking at your CI output you can see that rust dependencies are downloaded and rust is also compiling stuff.

The extractous rust library is doing (compiling) stuff with GraalVM because of Apache Tika, see extractous/extractous-core/README.md at main · yobix-ai/extractous · GitHub

I also don’t get it running locally, even with a newer rustler version with the deprecations fixed and a GraalVM Java SDK available.

It compiles without errors but also fails with the same Failed to load NIF library error.

Maybe someone has an idea what happens with the libtika_native.so file and how to handle it with a rustler setup.

@codestirring I think you are the article author? Can you tell us how you were able to get it running?

Thx

Cheers
Frank

snewcomer · January 31, 2025, 8:47pm

Appreciate the response! Oddly still failure to load with mix compile

ci.yaml
failure to load

snewcomer · January 31, 2025, 9:55pm

Great point. This might be specific to this rust package. I can’t figure this out either in CI. I can run it on my Mac but I haven’t discerned why that is.

suchasurge · February 1, 2025, 10:32am

Ah, ok I only tried on linux.

But I just build a simple rust CLI with extractous usage like in the article which works without complaining about libtika_native.so like with rustler.

So there might be some steps needed when used with rustler I can’t figure our right now.

snewcomer · February 2, 2025, 8:28pm

Yes I think you are right about rustler. Building it in Github Action is trying to reference that libtika_native.so from that /runner/ folder. At this point, I am at a loss.

[warning] The on_load function for module Elixir.NifExtractous.Native returned:
{:error,
 {:load_failed,
  ~c"Failed to load NIF library: 'dlopen(/Users/snewcomer/github/my_site/_build/dev/lib/nif_extractous/priv/native/libnif_extractous-v0.3.2-nif-2.17-aarch64-apple-darwin.so, 0x0002): Library not loaded: /Users/runner/work/nif-extractous/nif-extractous/target/aarch64-apple-darwin/release/build/extractous-1392444ace32bce9/out/tika-native/build/native/nativeCompile/libtika_native.dylib\n  Referenced from: <D02FBE3D-DA3C-3332-9BD2-76946358E9A3> /Users/snewcomer/github/my_site/_build/dev/lib/nif_extractous/priv/native/libnif_extractous-v0.3.2-nif-2.17-aarch64-apple-darwin.so\n  Reason: tried: '/Users/runner/work/nif-extractous/nif-extractous/target/aarch64-apple-darwin/release/build/extractous-1392444ace32bce9/out/tika-native/build/native/nativeCompile/libtika_native.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/work/nif-extractous/nif-extractous/target/aarch64-apple-darwin/release/build/extractous-1392444ace32bce9/out/tika-native/build/native/nativeCompile/libtika_native.dylib' (no such file), '/Users/runner/work/nif-extractous/nif-extractous/target/aarch64-apple-darwin/release/build/extractous-1392444ace32bce9/out/tika-native/build/native/nativeCompile/libtika_native.dylib' (no such file)'"}

filmor · February 3, 2025, 12:33pm

Rustler is not aware of the additional libraries that are copied in extractous’s build.rs. You will have to extend your build process to copy them from Rust’s target directory to priv/native/....

snewcomer · February 3, 2025, 3:09pm

Thanks @filmor! Anybody aware of a precedent in the rustler community that we could pull from for this extractous rust library?

snewcomer · February 4, 2025, 4:24am

A little more investigation…

 ldd _build/prod/rel/**/priv/native/libdoc_extractor.so

	linux-vdso.so.1 (0x0000ffff82483000)
	libtika_native.so => not found
	libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff82380000)
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff821d0000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffff8244a000)

So it is just that one libtika_native.so binary. Nothing else. Extending the build process seems like a good path here.

snewcomer · February 5, 2025, 4:42pm

Alright this ended up working. See README for what I had to do!

tpain · February 23, 2025, 3:59am

I have the same issue as you. I’d like to try your solution out, but not sure how to integrate it. Any pointers would be appreciated!

snewcomer · February 24, 2025, 4:03am

{:nif_extractous, github: "snewcomer/nif-extractous"},

NifExtractous.Native.extract_document(local_path)

christianRakete · March 4, 2025, 12:45pm

When I add {:nif_extractous, github: “snewcomer/nif-extractous”} to mix.exs, I get the following error when starting the app both locally on the Mac M2 and on the server:


== Compilation error in file lib/nif_extractous/native.ex ==
** (RuntimeError) Error while downloading precompiled NIF: the precompiled NIF file does not exist in the checksum file. Please consider run: `mix rustler_precompiled.download NifExtractous.Native --only-local` to generate the checksum file..

You can force the project to build from scratch with:

    config :rustler_precompiled, :force_build, nif_extractous: true

In order to force the build, you also need to add Rustler as a dependency in your `mix.exs`:

    {:rustler, ">= 0.0.0", optional: true}

    lib/nif_extractous/native.ex:4: (module)
could not compile dependency :nif_extractous, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile nif_extractous --force", update it with "mix deps.update nif_extractous" or clean it with "mix deps.clean nif_extractous"

Does anyone else experience the same problem and has a solution or idea?

snewcomer · March 4, 2025, 1:10pm

Hi Christian! Do you have

{:rustler, "~> 0.36.0", runtime: false}

in your mix file?

christianRakete · March 4, 2025, 1:21pm

Hi Scott!

{:rustler, ">= 0.0.0", optional: true}

or

{:rustler, "~> 0.36.0", runtime: false}

makes no difference either way.

snewcomer · March 4, 2025, 1:50pm

What about

{:nif_extractous, "~> 0.6.0"}

christianRakete · March 4, 2025, 2:45pm

This is how it works! Thank you very much!

christianRakete · March 5, 2025, 2:11pm

Hello Scott,

I’ve seen that you’ve been working hard. Is there an example Dockerfile by any chance? I’m having some issues with the deploy and I’m not sure if libtika_native.so is being found.

Exception in thread "main": java.lang.UnsatisfiedLinkError: Can't load library: awt | java.library.path = [.Djava.awt.headless=truefrom_raw ptr argumentFailed creating the graal native vm,  Failed creating the graal native from pointer,  ]
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.NativeLibraries.loadLibraryRelative(NativeLibraries.java:141)
        at java.base@23.0.1/java.lang.ClassLoader.loadLibrary(ClassLoader.java:108)
        at java.base@23.0.1/java.lang.Runtime.loadLibrary0(Runtime.java:916)
        at java.base@23.0.1/java.lang.System.loadLibrary(System.java:2066)
        at java.desktop@23.0.1/java.awt.image.ColorModel$1.run(ColorModel.java:211)

snewcomer · March 5, 2025, 2:29pm

Same - Stall when extracting using ocr on macos from pdf with embedded images · Issue #23 · yobix-ai/extractous · GitHub. Tried some other methods with that library

Here is the workaround I am doing

  defp extract_based_on_file_type(%{content_type: "application/pdf"} = _file, local_path) do
    case system_cmd().cmd("pdftotext", ["-layout", local_path, "-"]) do
      {contents, 0} ->
        {:ok,
         {contents,
          Jason.encode!(%{
            "content_type" => "application/pdf",
            "extracted_with" => "pdftotext"
          })}}

      error ->
        Logger.error("#{__MODULE__}.extract_based_on_file_type/2 error=#{inspect(error)}")
        {:error, :pdf_parse_failed}
    end
  end

  defp extract_based_on_file_type(_file, local_path) do
    # @todo error handling
    {:ok, NifExtractous.Native.extract_document(local_path)}
  end