Which GPUs can you use with Nx/Axon/etc?

Great question - I’ve been wondering the same!

Does anyone know if the built in GPU of a MacBook Pro would be ok or would I need to get an eGPU? My Mac has an AMD Radeon Pro 5500M 8 GB (and Intel UHD Graphics 630 1536 MB).

Related thread for external GPUs: Which external GPU are you using for Nx/Axon?

Doesn’t Nx backends use NIFs under the hood? For example the XLA backend there is a clear specification . The specifics of XLA backend is using CUDA so you most probably have to backtrack to that dependency, but this is only applicable to NVIDIA graphics cards. The way I see it there is still a big gap between official efficient universal drivers on unix systems, as graphics cards main market is currently gaming, witch is not existent on unix systems, this will change with time however this is a sensible topic as writing drivers for such kind of devices should take a lot of resources.

We definitely need more info on this, I can’t have been the only one thinking any OpenCL card probably would have been ok, even if not as performant as some of the others.

So basically it’s as @bdarla said, we need a modern Nvidia card?

Which NVIDIA cards are CUDA?

CUDA works with all Nvidia GPUs from the G8x series onwards, including GeForce, Quadro and the Tesla line. CUDA is compatible with most standard operating systems.

And from the site in the eGPU thread that’s these (for Mac users):

Screenshot 2022-12-12 at 22.26.361200×563 35.3 KB

How do you decide between them? Does the ‘additional software’ column matter if it’s just going to be for Nx?

XLA also supports AMD’s ROCm, but Nvidia/Cuda is the way to go?

AMD ROCm is the first open-source software development platform for HPC/Hyperscale-class GPU computing. AMD ROCm brings the UNIX philosophy of choice, minimalism and modular software development to GPU computing.

Which AMD cards support ROCm?

ROCm officially supports AMD GPUs that use following chips:

• GFX9 GPUs
  • “Vega 10” chips, such as on the AMD Radeon RX Vega 64 and Radeon Instinct MI25
  • “Vega 7nm” chips, such as on the Radeon Instinct MI50, Radeon Instinct MI60 or AMD Radeon VII, Radeon Pro VII
• CDNA GPUs
  • MI100 chips such as on the AMD Instinct™ MI100
  • MI200 such as on the AMD Instinct™ MI200
• RDNA GPUs
  • Radeon Pro V620
  • Radeon 6800 Workstation

so is there anything I have to do, like, on Linux, with AMD GPU to get the Nx working? I never touched it but may be interested.

As a beginner / student in this I like to see other practitioners’ information (I’m assuming some are on a low-budget and are thoughtful about what is worth spending money on).

One such video demonstrating benchmarking ML using an older graphics card vs a newer one is this one:

I’m in the same boat as you Hubert, but from my understanding there are some notable differences between ROCm and Cuda; ROCm is opensource but Nvidia Cuda cards are said to be more performant and have better compatibility/support (they’ve been around for 10 years).

AMD brought out GPUFORT as a direct competitor to Cuda last year:

This new GPUFORT project will live under the Radeon Open eCosystem (ROCm) umbrella and is their latest endeavor in helping developers with large CUDA code-bases transition away from NVIDIA’s closed ecosystem.

There is already HIPify and other efforts made by AMD the past several years to help developers migrate as much CUDA-specific code as possible to interfaces supported by their Radeon open-source compute stack. Most of those efforts to date have been focused on C/C++ code while GPUFORT is about taking CUDA-focused Fortran code and adapting it for Radeon GPU execution.

Source: https://www.phoronix.com/news/AMD-Radeon-GPUFORT

It also seems that ROCm doesn’t work on all AMD cards (even newer ones) and that AMD are still figuring out their game, whereas as most Nvidia cards (even consumer models) from the last 10 years or so are Cuda compatible, this has allowed Cuda to dominate the space… which makes Nvidia cards/Cuda the safe bet.

That’s the impression I’m left with anyway, maybe someone with more experience can let us know their thoughts

What’s the tldr David? Are newer cards better?

This AMD ROCm thing is wild, my old GPU that I just boxed month ago is supported and the new one is not? Pfff

TLDR
Whisper model finished in 1 minute on newer card vs 6 mins on older card vs 22 mins on CPU only
The guy is very impressed with his 12gb RTX 3060

I got to realise (from a bit of digging) that bigger RAM size is more important (to me) than incremental increase in GPU speed since, as per another comment, you can load larger models and get more accurate results - or can even complete the run of a large model ( viz on apple M1 max which has a ton of shared RAM running pytorch Running PyTorch on the M1 GPU | Sebastian Raschka, PhD)

But then this project sounds better - GitHub - bigscience-workshop/petals: 🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading · GitHub

TLDR shared access to GPUs to run big models

From the provided link on the “petals” project:

System requirements: Petals only supports Linux for now. If you don’t have a Linux machine, consider running Petals in Docker (see our image) or, in case of Windows, in WSL2 (read more). CPU is enough to run a client, but you probably need a GPU to run a server efficiently.

I am afraid using Apple Silicon based machines is not yet a promising direction (unfortunately). Nevertheless, thank you for sharing this information!