djaouen
I am trying to use Meta’s Llama 2 with Bumblebee, but I am getting a 401 error when I try to load it. I have been granted access to the repo on Hugging Face, but I think I need to provide an access token when loading the model, and I am not sure how to do so.
{:ok, model} = Bumblebee.load_model({:hf, "meta-llama/Llama-2-7b-chat-hf"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "meta-llama/Llama-2-7b-chat-hf"})
** (MatchError) no match of right hand side value: {:error, "HTTP request failed with status 401"}
(stdlib 5.0.2) erl_eval.erl:498: :erl_eval.expr/6
#cell:oqtu7kdr36by6ud4yug4rxbxs3qw544i:15: (file)
Where Next?
Trending in Questions
Other Trending Topics
Latest Livebook Threads
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #blog-post
- #phoenix_html
- #iex
- #graphql
- #ai
- #genstage
- #elixirconf-us
- #websockets
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #security
- #hex
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Bringing learning, work and compliance together since 1987.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 10 of 12 Posts
regex.sh
I don’t think Bumblebee supports that for now?
Does it? @seanmor5
EDIT:
Correction, there seems to be auth_token but for cached downloads
https://github.com/elixir-nx/bumblebee/blob/main/lib/bumblebee/huggingface/hub.ex#L41
djaouen
Thanks for looking into this for me. Is there a way to utilize this with
load_model,load_tokenizer, andload_generation_config?jonatanklosko
Hey @djaouen, you can specify it in repository options:
{:hf, "meta-llama/Llama-2-7b-chat-hf", auth_token: "..."}: )djaouen
Thanks, @regex.sh and @jonatanklosko!
haavars
Did you ever get i to work?
djaouen
Not yet. I was waiting for Bumblebee to get upgraded so that I could use
Bumblebee.Text.Llama.drewble
Looks like v0.3.1 has the official support for
Bumblebee.Text.Llama. Is that not working for you?djaouen
Actually, I don’t know what version of Bumblebee I was using, as it’s been long enough that Hugging Face deleted my notebook. But I will certainly look into that if I build out a similar project in the future. Thanks for the info!
nutheory
I was able to get “NousResearch/Llama-2-7b-hf” working using 0.3.1. Does anyone know if we can use GGML locally so i don’t have to wait for 5minute inference for two sentences on my M2?
josevalim
You can go ahead and do the bindings for llama.cpp yourself. Check out Erlang NIFs.
Otherwise, docs for LLaMA have been added here: https://hexdocs.pm/bumblebee/llama.html