Keep in mind that current autoregressive LLMs are heavily bottlenecked by memory bandwidth with low batch sizes (e.g. local inference). If you divide the memory bandwidth by the size of the model you get a reasonably accurate inference speed estimate.

So e.g. if you have a chip with 300GB/s bandwidth and you run a q4 70B model, about 35GB, you will get around 8 t/s. If you look at benchmarks for M chips that’s about right. Frankly 8 t/s feels pretty slow but it’s not unusuable.

However, if you were to jump to, say, a 100GB model, now you’re at 3 t/s. So even if you spend big on 128GB RAM there is not enough bandwidth to move those weights in and out of the registers and the performance is not good.

However, MoE models are designed to only load a subset of weights into the cores for each token. So if you have a 100GB model which only uses 10GB of weights for each token, now you have 30 t/s (pretty good) for your 100GB of RAM. This is why MoE models exist. Deepseek R1 (the real one) or LLama4 are in this category.

Anyway, the point being: these chips with 2-300 GB/s bandwidth might look nice, but keep in mind a 5090 is delivering 1800 GB/s memory bandwidth for its 32 GB of VRAM, i.e. generations will be 6-9 times faster flops notwithstanding (and it has plenty of flops too). The amount of RAM is not the only variable.

Also diffusion models might become a thing at any time and render this entire equation irrelevant, so who knows.

If you can wait 6 months or so, when the Mini M5’s come out (or a bit later for the Studios) you might be able to pick some up on the second hand market and build a cluster.

Worth keeping an eye on the AI portal on DT too, as it’ll usually show the most interesting threads in the trending lists: https://devtalk.com/ai

What about a cluster of M4 Minis?

Yes - clustering seems like an emerging thing for local inference. The NVIDIA DGX Spark is built to cluster. So is the Framework Desktop.

A popular framework for clustering is an open source tool EXO.