djaouen
I am trying to use Meta’s Llama 2 with Bumblebee, but I am getting a 401 error when I try to load it. I have been granted access to the repo on Hugging Face, but I think I need to provide an access token when loading the model, and I am not sure how to do so.
{:ok, model} = Bumblebee.load_model({:hf, "meta-llama/Llama-2-7b-chat-hf"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "meta-llama/Llama-2-7b-chat-hf"})
** (MatchError) no match of right hand side value: {:error, "HTTP request failed with status 401"}
(stdlib 5.0.2) erl_eval.erl:498: :erl_eval.expr/6
#cell:oqtu7kdr36by6ud4yug4rxbxs3qw544i:15: (file)
Where Next?
Trending in Questions
Other Trending Topics
Latest Livebook Threads
Latest on Elixir Forum
Sponsor Spotlight
Phoenix, LiveView, multimedia pipelines, distributed systems. 400 engineers, Elixir since 2017. Available for hire.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #library
- #deployment
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #elixirconf
- #channels
- #exunit
- #code-sync
- #discussion
- #podcasts
- #javascript
- #onsite
- #dialyzer
- #docker
- #authentication
- #ai
- #umbrella
- #elixirconf-us
- #full-time-contract
- #podcasts-by-brainlid
- #blog-post
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #graphql
- #genstage
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #elixirconf-eu
- #processes
- #security
- #api
- #forms
- #performance
Our Sponsors
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Elixir, Erlang & BEAM conferences from Code Sync. Next: ElixirConf EU 2027
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
**Building the future of software with AI, trust, and human connection.**
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
Showing Posts 1 to 10- Show Best Posts
- Show All (oldest first)
- Show All (newest first)
regex.sh
I don’t think Bumblebee supports that for now?
Does it? @seanmor5
EDIT:
Correction, there seems to be auth_token but for cached downloads
https://github.com/elixir-nx/bumblebee/blob/main/lib/bumblebee/huggingface/hub.ex#L41
djaouen
Thanks for looking into this for me. Is there a way to utilize this with
load_model,load_tokenizer, andload_generation_config?jonatanklosko
Hey @djaouen, you can specify it in repository options:
{:hf, "meta-llama/Llama-2-7b-chat-hf", auth_token: "..."}: )djaouen
Thanks, @regex.sh and @jonatanklosko!
haavars
Did you ever get i to work?
djaouen
Not yet. I was waiting for Bumblebee to get upgraded so that I could use
Bumblebee.Text.Llama.drewble
Looks like v0.3.1 has the official support for
Bumblebee.Text.Llama. Is that not working for you?djaouen
Actually, I don’t know what version of Bumblebee I was using, as it’s been long enough that Hugging Face deleted my notebook. But I will certainly look into that if I build out a similar project in the future. Thanks for the info!
nutheory
I was able to get “NousResearch/Llama-2-7b-hf” working using 0.3.1. Does anyone know if we can use GGML locally so i don’t have to wait for 5minute inference for two sentences on my M2?
josevalim
You can go ahead and do the bindings for llama.cpp yourself. Check out Erlang NIFs.
Otherwise, docs for LLaMA have been added here: https://hexdocs.pm/bumblebee/llama.html