I don’t know about them breaking down but Gemini Pro starts showing laziness tendencies when the chat nears the 1M tokens context window. It starts hypothesizing on code it has readily available and given to it in the initial prompt and I have to keep reminding it that it already has the code and it should just check it.

I think you are anthropomorphizing here. Not only is it trivial for a computer to memorize Hamlet, but there are human savants who can do so with ease as well. It seems to be more a matter of evolutionary pressure: it was not evolutionarily useful for the average human to memorize Hamlet. And even this is a relative matter; a monkey would have great difficulty memorizing the poem.

But this discussion is somewhat moot in that it is far too expensive for models to actually operate natively with 1M context windows. My understanding is that large context models use tricks internally which, with some hand waving, are conceptually similar to RAG in that they compress the internal representation of the tokens into a latent vector. In other words, the RAG is coming from inside the model.

The only question is, empirically, which works better. I can’t speak to this because I have not been vibecoding, but it sounds like you (and others) are saying that the large context models don’t work well enough at this time.

I imagine there is also a cost factor, as input tokens are not free. But I’m sure the providers would love if you would send them more!

As far as I can tell, this is a key mistake. RAG built upon AST can keep relevant context for nearly billions of tokens, on the contrary to “large” models which still need to operate the context on their side.

That’s exactly the reason Ragex was born. I felt like I hit the ceiling too fast, exactly because the context window cannot grow infinitely, no matter what providers say. With Ragex, I’ve tried the elixir codebase and it somewhat works without any degrading, because AST is way more intense, compared to plain text.