Can't run model via bumblebee: (RuntimeError) conversion failed, expected "eos_token_id" to be a number

Elixir Framework Forums Nx Forum
, ,
1

Hi, I am a web dev and complete noob in ML stuff. I have a use case to generate SQL query from text prompt and was trying to run this model using bumblebee.

I don’t understand much of the ML terminologies and was trying to build something from the example code on the bumblebee hex docs.

This is what I wrote

repo = {:hf, "defog/llama-3-sqlcoder-8b"}
{:ok, sqlcoder_model} = Bumblebee.load_model(repo)
{:ok, tokenizer} = Bumblebee.load_tokenizer(repo)
{:ok, generation_config} = Bumblebee.load_generation_config(repo)

serving = Bumblebee.Text.generation(sqlcoder_model, tokenizer, generation_config)
prompt = "Write a SQL query to count all rows"
Nx.Serving.run(serving, prompt)

I face an error like

** (RuntimeError) conversion failed, expected "eos_token_id" to be a number, got: [128001, 128009]

I don’t understand what this means, can anyone help or point me in the right direction, here? Thanks

1 Like
2

Hey @arpan, I believe this has been fixed on Bumblebee main. You can install it as {:bumblebee, github: "elixir-nx/bumblebee"} to try : )

1 Like