Hello! I’m following the example here bumblebee/fine_tuning.livemd at main · elixir-nx/bumblebee · GitHub for fine tuning the BERT model from huggingface using bumblebee, and I keep running into memory issues. I’m setting up a Tesla T4 GPU on GCP (posted my setup steps here: Getting set up to do GP…

Did you alter the livebook code in any way? Especially the sequence length and batch size parameters?

Thanks for your reply! No, pure copy and paste except for the CSV file path. It also failed on my macbook pro when doing cpu training, which is what led me down the path of provisioning a gpu.

It’s odd to me that it’s failing then. The batch sizes are small, and it seems that the BEAM process is fluctuating around 5GB RAM running on the CPU. Your GPU has 14GB available it seems, so I would expect that to work properly. I recommend that you open an issue on GitHub because this does smell…

Thanks :pray:, I’ll open an issue on Bumblebee

for what its worth I watched the mem usage with a cpu backend for a while and saw it spike to 32GB used in the second epoch… maybe I just underestimated how memory hungry this training process is.

Ah, try forcing versions 0.5.3 instead of 0.5.1 just for the sake of the sanity check please?

No difference with 0.5.3 :frowning:

I was able to get this fine tuning example working with the medium BERT model from hugging face. Here is the full source if you want to see that in action For those curious to learn more about the full setup and even what the feedback loop is like between CPU/GPU :slight_smile: I wrote a blog about…

Elixir Programming Language Forum

Memory issues following Bumblebee example

Elixir Framework Forums Nx Forum

wrgoldstein April 20, 2023, 12:59pm 5

Thanks , I’ll open an issue on Bumblebee