Using "chentong00/propositionizer-wiki-flan-t5-large" with Bumblebee?

Elixir Framework Forums Nx Forum
,
1

Would it be possible to use Elixir libraries (e.g. Bumblebee) to load and use the propositionizer model? How would one be able to generate an example in pure Elixir as the one provided by the author of the model in Python?

This model generates/proposes “atomic” segments of an initial input with a relatively complete context. It can be useful for RAG applications.

The corresponding paper can be found here.

Thanks in advance for any relevant feedback!

2

Hey @bdarla, here is a notebook with a corresponding example in Bumblebee:

# Propositionizer

```elixir
Mix.install([
  {:bumblebee, "~> 0.5.3"},
  {:exla, "~> 0.7.3"},
  {:kino, "~> 0.13.1"},
  {:jason, "~> 1.4"}
])

Nx.global_default_backend(EXLA.Backend)
```

## Setup the serving

```elixir
repo = {:hf, "chentong00/propositionizer-wiki-flan-t5-large"}
tokenizer_repo = {:hf, "google/flan-t5-large"}

{:ok, model_info} = Bumblebee.load_model(repo)
{:ok, tokenizer} = Bumblebee.load_tokenizer(tokenizer_repo)
{:ok, generation_config} = Bumblebee.load_generation_config(repo)

generation_config = Bumblebee.configure(generation_config, max_new_tokens: 512)

serving =
  Bumblebee.Text.generation(model_info, tokenizer, generation_config,
    compile: [batch_size: 1, sequence_length: 512],
    defn_options: [compiler: EXLA]
  )

Kino.start_child({Nx.Serving, serving: serving, name: Propositionizer})
```

<!-- livebook:{"output":true} -->

```
{:ok, #PID<0.264.0>}
```

## Inference

```elixir
title = "Leaning Tower of Pisa"
section = ""
content = "Prior to restoration work performed between 1990 and 2001, Leaning Tower of Pisa leaned at an angle of 5.5 degrees, but the tower now leans at about 3.99 degrees. This means the top of the tower is displaced horizontally 3.9 meters (12 ft 10 in) from the center."

input_text = "Title: #{title}. Section: #{section}. Content: #{content}"

%{results: [%{text: output_text}]} = Nx.Serving.batched_run(Propositionizer, input_text)
```

<!-- livebook:{"output":true} -->

```
%{
  results: [
    %{
      text: "[\"Prior to restoration work performed between 1990 and 2001, Leaning Tower of Pisa leaned at an angle of 5.5 degrees.\", \"Leaning Tower of Pisa now leans at about 3.99 degrees.\", \"The top of Leaning Tower of Pisa is displaced horizontally 3.9 meters (12 ft 10 in) from the center.\"]",
      token_summary: %{input: 84, output: 84, padding: 428}
    }
  ]
}
```

```elixir
Jason.decode(output_text)
```

<!-- livebook:{"output":true} -->

```
{:ok,
 ["Prior to restoration work performed between 1990 and 2001, Leaning Tower of Pisa leaned at an angle of 5.5 degrees.",
  "Leaning Tower of Pisa now leans at about 3.99 degrees.",
  "The top of Leaning Tower of Pisa is displaced horizontally 3.9 meters (12 ft 10 in) from the center."]}
```
2 Likes
3

Wow, thank you so much Jonatan! I cannot express my gratitude enough!

1 Like