Hey @sehHeiden, you need start_link, rather than start_child. Also, to specify the batch size, pass :compile to the serving configuration, so the computation is compiled on serving startup.

{:ok, model_info} = Bumblebee.load_model({:hf, "finiteautomata/bertweet-base-sentiment-analysis"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "vinai/bertweet-base"})

english_sentiment_serving =
  Bumblebee.Text.text_classification(model_info, tokenizer,
    compile: [batch_size: 16, sequence_length: 130],
    defn_options: [compiler: EXLA]
  )

children = [
  {Nx.Serving, serving: english_sentiment_serving, name: EngSentimentServering}
]

Supervisor.start_link(children, strategy: :one_for_one)

When in Livebook, the best way to start the serving is Kino.start_child({Nx.Serving, ...}).

that I start a Genserver with the model,tokenizer on each of the 16 cores

As single serving process is started. Tokenization happens in the calling process to get raw input, the raw inputs are sent to the server and batched, then the model runs for that batch and utilizes multiple cores (how exactly is up to the XLA compiler).

Also note that this particular model accepts max input of 130 tokens (model_info.spec.max_positions), so longer texts need to be truncated (which we ensure by compiling for sequence_length: 130 above).

@sehHeiden ah yeah, I fixed this on main, please try {:bumblebee, github: "elixir-nx/bumblebee"}.

It should behave like this in Nx v0.6.1.