sehHeiden
Batched_run for bumblebee Model
want to do sentiment analysis and language predictions on text with a livebook. Both are available with bumblebee.
I am hitting the problem that the number of text is to large for the ram. In addition I am wondering why Nx.Serving.run only uses a single batch. Therefore I looked into the Nx.Serving documentation and thought that Nx.Serving.batched_run was the way to go. I tried to follow the documentation an d came up with:
{:ok, model_info} = Bumblebee.load_model({:hf, "finiteautomata/bertweet-base-sentiment-analysis"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "vinai/bertweet-base"})
english_sentiment_serving =
Bumblebee.Text.text_classification(model_info, tokenizer,
defn_options: [compiler: EXLA]
)
children = [
{Nx.Serving,
serving: english_sentiment_serving,
name: EngSentimentServering,
batch_size: 16}
]
Supervisor.start_child(children, strategy: :one_for_one)
So what I thing I do is, that I start a Genserver with the model,tokenizer on each of the 16 cores. What I do get is an Error that Genserver.whereis got the wrong type of arguments.
Most Liked
jonatanklosko
Hey @sehHeiden, you need start_link, rather than start_child. Also, to specify the batch size, pass :compile to the serving configuration, so the computation is compiled on serving startup.
{:ok, model_info} = Bumblebee.load_model({:hf, "finiteautomata/bertweet-base-sentiment-analysis"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "vinai/bertweet-base"})
english_sentiment_serving =
Bumblebee.Text.text_classification(model_info, tokenizer,
compile: [batch_size: 16, sequence_length: 130],
defn_options: [compiler: EXLA]
)
children = [
{Nx.Serving, serving: english_sentiment_serving, name: EngSentimentServering}
]
Supervisor.start_link(children, strategy: :one_for_one)
When in Livebook, the best way to start the serving is Kino.start_child({Nx.Serving, ...}).
that I start a Genserver with the model,tokenizer on each of the 16 cores
As single serving process is started. Tokenization happens in the calling process to get raw input, the raw inputs are sent to the server and batched, then the model runs for that batch and utilizes multiple cores (how exactly is up to the XLA compiler).
Also note that this particular model accepts max input of 130 tokens (model_info.spec.max_positions), so longer texts need to be truncated (which we ensure by compiling for sequence_length: 130 above).
jonatanklosko
@sehHeiden ah yeah, I fixed this on main, please try {:bumblebee, github: "elixir-nx/bumblebee"}.
josevalim
It should behave like this in Nx v0.6.1. ![]()
Popular in Questions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance








