Oh, you mean the model repeats words, not that it gets stuck without output. So that is not a bug, it’s just how the model behaves under the given configuration.

There are several factors that determine the model output. First of all, sometimes the model has a separate “base” and “instruct” checkpoints. Base models are trained on a bunch of text to get “text understanding”, and instruct models are further fine-tuned on conversations and tasks, to make them more usable in a chat-like interaction. For example, there is HuggingFaceTB/SmolLM-1.7B and HuggingFaceTB/SmolLM-1.7B-instruct, you probably want to use the latter.

Next, you want to make sure you use a prompt relevant for the given model. Instruct checkpoints usually have a certain template for specifying the conversation history. With the transformers Python library you can specify a template on the tokenizer, however it use a Python-specific template, so we can’t reliably load it. However, you can find the template by looking for “chat_template” in tokenizer_config.json in the given repo. Sometimes the template is also present in the model readme. Continuing with SmolLM instruct, the template is here. So here is the prompt with template:

prompt = """
<|im_start|>user
Complete the paragraph: our solar system is<|im_end|>\
<|im_start|>assistant
"""

It results in a much better result.

Finally you can make the output non-deterministic (and more creative) by using a sampling strategy for the generation, such as this:

generation_config =
  Bumblebee.configure(generation_config,
    strategy: %{type: :multinomial_sampling, top_p: 0.6}
  )

(Sidenote: there is also :no_repeat_ngram_length to explicitly avoid repetitions, however it is a trade-off, because sometimes there are longer pharses, city names, etc, and preventing the model from reusing them worsens the usability)