So I was trying to migrate some python code that I was running into my elixir code. My set up is that I have Elixir running most of the code, and rustler/rust bindings that run rust-bert for one model, and some python code that runs for another one. My goal was to bring as much as possible into Elixir. I found a model that I could run in Elixir, that was performing better than the one in Python, but there seemed to be a crossing of wires since both the Rust code, and the Elixir Nx code were using PyTorch. It seemed like whichever one ran first would get the lock on PyTorch, and the second one wouldn’t work(on the Rust side i’d see it was a error like this: Nif not loaded ... symbol not found in flat namespace '__ZN3c1019UndefinedTensorImpl10_singletonE'.

When I tried to instead run the Elixir side on Exla, I could get it to work breifly in a completely clean project, but not all together with everything else. When I load the model I get this output:

[info] TfrtCpuClient created.
[debug] the following PyTorch parameters were unused:

  * bert.embeddings.position_ids
  * bert.pooler.dense.bias
  * bert.pooler.dense.weight

Which seems strange because I’m trying to target EXLA, not PyTorch. To confirm right before I ran this operation, I ran the following:

iex(3)> Nx.default_backend
{EXLA.Backend, [client: :host]}

Though the tensors do state EXLA:

f32[768]
EXLA.Backend<host:0,...

The model is started as follows:

{:ok, model} = Bumblebee.load_model({:hf, "vblagoje/bert-english-uncased-finetuned-pos"})
  # This is only coming from local because there's no fast tokenizer "tokenizer.json" file on HF, but loading it in Python, then dumping to a file created one.
  {:ok, tokenizer} = Bumblebee.load_tokenizer({:local, "priv/tokenizer"})
  model = Bumblebee.Text.token_classification(model, tokenizer, aggregation: :same)

And called like:

Nx.Serving.run(model, "This is a test")

Which gives me the following:

libc++abi: terminating with uncaught exception of type std::out_of_range: Span::at failed bounds check
[1]    67025 abort      iex --erl "-kernel shell_history enabled" -S mix phx.server

I have tried:

• Trying a different model, where both model and tokenizer are directly from HF, same result
• Scorched earth removal of pytorch, tensorflow, all deps, _build, the works, and then rebuilding, same result
• Completely removing rustler or anything else. Even trying in a completely clean project with just nx/bumblebee/axon/exla/jason, to try it there, same result.
• restarting the computer, same result.

I’m not even sure how to debug where it’s crashing from.

Here are my mix.exs deps:

      {:axon, "~> 0.5.1"},
      {:bumblebee, "~> 0.2.0"},
      {:nx, "~> 0.5.1"},
      {:exla, "~> 0.5.1", sparse: "exla"},
      {:jason, "~> 1.0"}

Anyone have an insight or tips of what I could try?