sehHeiden

Create ONNX files. Howto use imported model?

Hi,

wanted to do a sentiment analysis using Elixir. Problem: Bumblebee has that only for English text.

Therefore, tried to load a [model] (oliverguhr/german-sentiment-bert · Hugging Face) from python and Pycharm and chatpgt tried to export the tokenizer and model with:


# Initialize the model
model = germansentiment.SentimentModel()

# Dummy input that matches the input dimensions of the model
dummy_input = torch.randint(0, 30_000, (1, 512), dtype=torch.long)

# Export to ONNX
torch.onnx.export(model.model, dummy_input, "german_sentiment_model.onnx")

# Export the vocab
with open('vocab.json', 'w') as f:
    json.dump(model.tokenizer.vocab, f)

With this I was able to export the model and the vocabulary. Now I try, infer in Elixir using Nx and Axon_onnx:

{model, params} = AxonOnnx.import("./models/models/german_sentiment_model.onnx")

{:ok, vocab_string} = File.read("./models/models/vocab.json")
{:ok, vocab_map} = Jason.decode(vocab_string)

# Tokenize
input_text = "Ein schlechter Film"
token_list = Enum.map(String.split(input_text, " "), fn x -> vocab_map[x] end)
token_tensor = Nx.tensor(List.duplicate(0, 512 - length(token_list)))
token_tensor = Nx.concatenate([Nx.tensor(token_list), token_tensor])

{init_fn, predict_fn} = Axon.build(model)

predict_fn.(params, token_tensor)

The output is:

#Nx.Tensor<
  f32[1][3]
  EXLA.Backend<host:0, 0.1233469648.4027973659.33323>
  [
    [-1.17998206615448, 5.767077922821045, -5.835022926330566]
  ]
>

I assume from the weblink, that the zeroth argument is positive, the first is negative and the last is neutral. Just the scale is off. I assumend a sum of one.

Because this is export and use in elixir is a lot of first times for. I would start, with asking:

Did I do it right?
Would I do a cross entropy on the output?
Anything I could enhance?

8 comments

/nx #axon #axon_onnx

1 1666 8

2023-09-06 20:18:18 UTC

Marked As Solved

jonatanklosko

Creator of Livebook

Hey @sehHeiden, here’s a complete example in Bumblebee:

# German sentiment analysis

```elixir
Mix.install(
  [
    {:bumblebee, "~> 0.3.1"},
    {:exla, "~> 0.6.0"}
  ],
  config: [nx: [default_backend: EXLA.Backend]]
)
```

## Prediction

```elixir
{:ok, model_info} = Bumblebee.load_model({:hf, "oliverguhr/german-sentiment-bert"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "bert-base-german-cased"})

serving =
  Bumblebee.Text.text_classification(model_info, tokenizer, defn_options: [compiler: EXLA])
```

```elixir
texts = [
  "Mit keinem guten Ergebniss",
  "Das ist gar nicht mal so gut",
  "Total awesome!",
  "nicht so schlecht wie erwartet",
  "Der Test verlief positiv.",
  "Sie fährt ein grünes Auto."
]

Nx.Serving.run(serving, texts)
```

Tokenizer details

There are two ways in which tokenizers can be stored on HF Hub. It’s either (1) tokenizer_config.json + vocab.txt + optional merges.txt (this is a dump of a “slow” tokenizer from hf/transformers), or (2) a single tokenizer.json file (this is a dump of a “fast” Rust tokenizer from hf/transformers). Oftentimes the repository includes both versions. In Python, hf/transformers have a logic to load (1) and convert to a fast tokenizer, but we always rely on tokenizer.json, which we hand to the underlying Rust library. When a repository doesn’t have tokenizer.json, it is usually possible to find another base repository with the same tokenizer, which does have that file. In this case I looked at their training code (ref), they fine-tune bert-base-german-cased, so they use the same tokenizer and we can load it from there just fine.

Post #7

Also Liked

josevalim

Creator of Elixir

Take a look at defn (numerical definitions). Those a functions where you can write numerical code that works with tensors using the regular Elixir operators. You would get something like (untested):

import Nx.Defn

defn political_score(predict_fn, params, token_tensor) do
  prediction = predict_fn.(params, token_tensor)
  one_hot = 2 ** prediction / Nx.sum(2 ** prediction)
  5 * (one_hot[0][0] - one_hot[0][2])
end

Post #4

jonatanklosko

Creator of Livebook

As I ave the the vocab.json, is the the same as the tokenizer.json?

tokenizer.json is a single file with all information, other than vocabulary, it includes special tokens information, tokenizer model, pre/post processing, etc. See tokenizer.json for an example.

Which Rust library is used in Elixir to load it?

There is huggingface/tokenizers in Rust and it also has Python bindings. huggingface/transformers have two types of tokenizers, slow - implemented purely in Python and fast - calling out to the Rust library. We have elixir-nx/tokenizers with bindings to the Rust library.

Can you assume, why the onnx model has another output scale than the original/bumblebee version?

I may be missing something, but I don’t think the calls are equivalent. You are splitting on space and using the vocab. The tokenizer on the other hand does more, it will split longer words into parts and add special tokens.

Post #9

Last Post!

jonatanklosko

Creator of Livebook

As I ave the the vocab.json, is the the same as the tokenizer.json?

tokenizer.json is a single file with all information, other than vocabulary, it includes special tokens information, tokenizer model, pre/post processing, etc. See tokenizer.json for an example.