I was referring to the main theme of this thread, which is the forced deprecation of writing code by hand at work, before getting up to a level of proficiency.
derek-zhou
dimitarvp
Ah, thank you, and yes I of course 100% agree with this. This current phase in the human civilisation (and IT area as well) just demonstrates how clueless so many are that they still think the problem was that we the programmers were some elitist gatekeepers who deliberately stopped businesses from succeeding when, you see, and all we ever had to do was to “just ship it”. 
They’re hopeless. Turns out that nuanced thinking is a rare skill out there.
I see no way out outside of us cooping up together and throwing each other work and make and maintain mutual support networks, and just generally find multiple ways to weather the storm. Because it’ll end at one point – but I admit I am at my wits end if I ask myself the question “But then what?”.
5
budgie
Yes, 1000%. It’s crazy to me. It’s like nobody does anything without it anymore, and it’s depressing. I mean long before ChatGPT, we had developers pushing out MVPs in a weekend.
People are publishing AI crap by any means necessary. It’s important, but not as important as product/engineering/marketing/sales theory. The why behind everything matters a lot more than how we get there today
.Why do you think there are so many “AI researchers/experts” pushing the opposite? Is it just personal gain, really deluded, or are they not actually experts?
This is what makes me feel gaslit the most.
It still doesn’t make sense to me to say you can generate more correctness than what you can already generate in a chat completion. Which is necessary for recursive self improvement, otherwise you’re just decompressing information already available. Could be useful to boost representation for the more correct parts of the model that are currently underrepresented, but overall intelligence gains are highly unlikely.
2
jeremyjh
The sentence you are quoting is just explaining what post-training is. The same models are used for chat and agentic coding, and they are trained in many, many ways that accumulate different behaviors. ChatGPT 3 was initially largely post-trained with human feedback, and that is what made it different from some of the auto-complete solutions we had before that - those truly were just predicting the next token based on the statistical likelihood inferred from the internet corpus. If that is your understanding of how coding models work today, then you are very fundamentally mistaken.
Most of the performance gains in the past eighteen months are due not to larger models and training sets, but to further improvements in architecture, harness and training methods. I’m not sure if you know very much about deep learning, but the reason AlphaGo and AlphaZero / Leela Chess Zero were able to become absolutely monsters that outperformed every human is because they played millions of games against themselves and the successful moves they played - that satisified the reward signal of winning - reinforced weights that went into the next round. The success in these domains happened so early because there is very little cost to playing milions of games this way, and the only content they started with was an environment that enforced the rules of the game. All the information content came from self-play.
The same thing is happening now with coding agents and RLVR; the models can spend thousands of subjective years solving math problems, or implementing coding solutions in a harness environment, and the successful moves reinforce the model weights in ways that we have no understanding or control over.
This is the point, is we do get new information content and signal from what happens in RLVR environments.
Our only job in that phase of research is to supply them with novel problems and the environments in which to solve them - and then their struggle for the solution itself is where the information required for the extra capability is coming from. We get a tremendous amount of leverage with these methods because very little human thought and work has to go into setting up the exercise, compared to the amount of novel input a model can get from spending hours trying to solve it.
So, that is why they are still getting better. But my main point is that once we are training this way, they are not “predicting tokens” in a meaningful way, they are predicting “satisfies reward function”. And in some cases, they are making that prediction about approaches that will take them many hours and many turns to develop, in advance.
2
Desty
Well, AI is (or… was, before everything collapsed into genAI) a huge field, with lots of different approaches, applications and philosophies. My doctorate was focused on only one small aspect of it, which is often the case, so I wouldn’t consider myself an expert (in anything).
I don’t hang around with LLM-focused academics, since I finished my PhD long before ChatGPT came out, but many of them are probably vulnerable to the same “there’s no going back now” fallacious arguments as the rest of us. That’s why, when people make those arguments, I don’t think they’re stupid… I just don’t agree. Greedy employers have always wanted to cut corners, fire staff and keep more money for themselves… LLMs just happened to be a very convincing sales pitch for exactly that kind of short-term greedy mindset.
Actually I’d rather see those employers at least being honest that they’ll cut you loose to make some quick money, compared to the old days where they’d say “we’re a family” while axing staff. At least now the mask is off and nobody has to pretend. But I think maybe (just maaaaybe) they’re starting to discover that short-term wins from forcing their engineers to speedslop might be cancelled out by long-term damage to quality and maintainability.
Either that happens, or I’ll have to become a piano teacher, or actually build that startup SaaS I’ve been talking about for years but never doing 
3
i-n-g-m-a-r
Thank you for writing this. It’s very important, at least to me it is. Except for some “brainstorming” and poking around out of curiosity I’ve never used any LLM to do any work for me. I was also among the very last “pockets of resistance” to eventually purchase a mobile phone in the Nokia era. Just to give you an idea.
I’d like to comment on your general feeling. Just to give you some food for thought. I think what is called “information age” should be termed Dunning-Kruger age. You are rejecting what to think. You are questioning. Now you are at odds with “the system”.
By extension you are now also at odds with many of your peers and with the general public. You feel sad about this so you must be a beginner. I can assure you feeling sad is just a phase. You will find what you love on the other side. You may rediscover passions lost or find new challenges to engage with.
Of course none of this may apply to you, just sharing my thoughts.
3
fullpin
I read through this whole thread and it’s been a great read! I share the feelings and perspectives of many people here. I don’t usually post (the dark matter dev in me) but this one really dragged me in haha.
I am certainly no expert in LLMs, and I don’t want to straw man if you were responding to a specific context but I have a few comments/questions on this post.
I’m not sure if you know very much about deep learning, but the reason AlphaGo and AlphaZero / Leela Chess Zero were able to become absolutely monsters that outperformed every human is because they played millions of games against themselves and the successful moves they played
I think this is a straw man. I’m assuming you and most know all this but wanted to mention to illustrate a later question::
there were also chess engines that could beat people way before using ‘symbolic AI’ (DeepBlue, stockfish), they were basically:
- make tree of all of the possible game moves in a smart way
- pick the next move that gives you the best value based on the tree (human defined)
Then alpha zero came in and beat these engines based on non-symbolic AI and RL using win/draw/loss reward function. All of this to say, the actual improvement was beating a different engine.
The same thing is happening now with coding agents and RLVR; the models can spend thousands of subjective years solving math problems, or implementing coding solutions in a harness environment, and the successful moves reinforce the model weights in ways that we have no understanding or control over.
My question on this is: what is the reward function? If we are just talking code works vs code no work or code fast vs. code slow then yes makes sense. Do you mean this in a broader sense, like ‘code good’?
Also, I think @budgie is talking at a different level here – recursive self improvement in the context of it becoming AGI, or replacing all work, by being trained on it’s own (or others) outputs. Specifically what is the reward function for any of the fuzziness in our human world and roles, where I think higher-level software development/engineering/architecture/whatever still has lots, and which is largely where (IMO) the neural net/transformer and corpus kicks in and starts thinking for you, which can be helpful sometimes and hurtful others.
My biggest critique is it can be a very slippery slope in my experience to go from ‘I just use it for a, b and c’ to I use it for ‘a, b c, … z’.
2
jeremyjh
We also learned new things about chess from AlphaZero, such as the long-term value of early rook pawn pushes (and many more). Stockfish couldn’t teach us anything new since we had already written the rules it used for evaluation. My point isn’t that the chess engine is a miracle but rather a deep learning model can learn from self-play, it doesn’t have to consume existing information that someone has prepared for it.
The reward function is whatever is defined for the exercise its doing. So yes, this is at the most basic level “does the code work”. But doing the work necessary to satisfy this involves MANY bits of new information. This is also true of new human programmers banging their head against the keyboard until they understand it is their code that is broken and not the compiler.
Maybe so, but I wasn’t responding to that. I was responding to this statement:
they have no underlying models of anything other than “what token is likely to appear next”.
And all I pointed out was, that this statement is incorrect. And anyone who - in 2026 - works in technology and still doesn’t understand this really should update.
0
caleb-bb
I agree.
There’s a limit up to which AI can do things, and that limit is wherever humans stopped. Recursive self-improvement for current AI means AI can, by itself, make stuff exactly as good as what has been reliably mass-produced, and no more. ¯\_(ツ)_/¯ that’s how it looks to me anyway.
Of course, we can get some insight from watching LLMs work. But I think model collapse shows there’s some kind of radical creativity that deep learning models still lack. And I’m guessing recursive self-improvement, if we pull it off, means “AI can, given novel material, take itself up to the limit where model collapse happens”. So it hits its ceiling immediately, in other words.
0
budgie
I know it’s not just pre-trained text in a chat interface. But either way it’s optimized for user desire because there is no notion of correctness that’s immediately available in the vast majority of real world applications.
That’s why it’s different with programming, because the result is immediately verifiable through just running the program or even through automated tests.
RLHF is also very faulty because it’s a yes/no for a whole generation. There’s no context or nuance you can give to teach it what specifically is right or wrong about a generation. You just nudge all weights in either direction.
You want anything else, you’re going to need human written pairs for fine-tuning. But as we know, that doesn’t scale.
RLVR is just not possible in any meaningful way in large models for a lot of context/project/domain-specific work. This is why I believe small models are the way forward.
The knowledge about and ease of training and fine-tuning small models needs to get a lot better before I see this changing.
As I mentioned, not comparable at all to free-form text generation imo. No matter the complexity, chess and go are closed-form problems and evaluating the correctness in a loop is as simple as looking at the rules.
Yes exactly. I want to clarify that we can still make advancements with AI through cross-context applications of knowledge. But that doesn’t mean we magically get AGI or superintelligence.
0