One thing to keep in mind is that small local models are not useless, they are great fuzzy logic implementations or great text wranglers.

I use Qwen through ollama on my MacBook Air for various little tasks : receipts classification (my accountant is very old school), enrichment of transcripts (transcribed locally with whisper) with project context, and data structuring from quick text notes. The scripts that run those tools were also partly written locally by qwen !

If the Talaas exploration (a startup that burns LLM weights on silicon) succeds, little specialized models might be way more common. Llama8b-in-an-USB-stick running at 10000 tokens/second can be very useful. (If you did not try this demo, https://chatjimmy.ai/ demonstrates supposedly on-chip inference of llama8b, my tries hover around 15000 tokens/second).

So small local models seem to be on a track to become « edge inference » with lower consumption… still bringing another new generation of hardware and gadgets though .

The problem I see here is a “misalignment of expectations” (within the AI retail market - i.e. us). The NVIDIA’s market is essentially data centers. Why would they jeopardize their customers’ revenues from token sales with a product they could only sell to us every once in a while? IMO, their strategy so far has been going in the opposite direction - to finance their customers’ purchases of their own products en mass (it hardly gets any bubblier than that, btw).

A rough breakdown of NVIDIA’s sales for the last fiscal year (according to Grok):

• Total revenue: $215.9 billion (up 65% YoY)
• Data Center: $193.7 billion → ≈ 89.7%
• Gaming (GPUs for the gaming industry): $16.0 billion

Also, the LLM providers, what incentive would they have to train models we can then use for free? That’s not the name of the game.

Maybe, just maybe, this opens a new window of entrepreneurial opportunity: training and selling models for private use, but again, with the customers running them on what?

Thinking about all the AI models companies like Google, Meta and others have made available for free it seems to me that is part of the game. For most training some of the big models (not just LLMs) is impossible due to both lack of enough data and processing resources. But here freebies had made a lot of difference as fine tuning of such available models is achievable for many. Either way there are free LLMs available today too, and I hope they will keep improving as well.

I think that the commercial big ones will likely always be better. But all LLMs keep improving, while the difficulty of programming stay fairly constant if not easier. Thus my hope is that at some point local LLMs will make production quality code. Then I will ask myself if paying a service to do even better quality or faster is really worth it. Good enough is good enough at some point.

I don’t know. At some level this huge LLMs providers in the sky remind me of laser printing services. Great businesses until the prices and quality was such that every office just got their own laser printer instead. I think LLMs for programming might be temporary. Will we still be doing it like we are now in 5 to 10 years? I’m not so sure.