Ash Framework: Official LLM development tooling and guidance

Actually I think this could be an opportunity for Ash to shine as its declarative model makes it perfect to generate high quality code using LLM that could provide even faster starting point to bring products live

I concur.

I’m curious, as the author of a relatively niche framework, and therefore as someone who of course knows Ash as well as it can be known: are current models capable of generating “useful” Ash code in your view?

There is some concern (including from myself) that if model-assisted programming does take off there will be a “winner takes all” economy where anything niche (perhaps Elixir itself) is squeezed out.

One thing I found interesting (and concerning) was the recent Camel research published last month, which detailed a potential solution for prompt injection attacks. The work itself is a big deal because it’s the first seemingly working strategy to deal with such attacks, but the concerning part is the authors elected to create a DSL with Python syntax to model the permission system.

That is to say, the work had nothing to even do with Python and yet they still felt the need to create something which looks like Python just to improve the results.

On the other hand, it sounds like giving models clear instructions on how to use Elixir and Ash can be helpful. Empirically, can anyone share how much of a difference these things make?

My personal and subjective experimation shows a night and day difference with:

• good rules files
• providing reference for whatever you’re making

As for the winner takes all, it’s possible, but only if models don’t get good enough to do more than just parrot popular tools, at which point who cares because its not going anywhere .

Models in their current form are always going to be better at parroting existing tools because that is how they’re designed. This is not to say they don’t exhibit a form of intelligence, but there is a substantial advantage for e.g. Python over e.g. Elixir because the model simply “knows” Python better.

That’s why I think the example I gave is so interesting: they were implementing a DSL, they could have chosen any syntax, so they went with Python knowing that the model would perform a bit better with it.

On the other hand, it could be (as discussed above) that something like Ash hits an abstraction sweet spot where, perhaps, models are able to produce more maintainable code due to the constraints given. Similar to the benefit for a human! It will be interesting to see if things play out this way.

If models were good enough to learn any tool with no training bias I think we would be in superintelligence territory, and at that point none of this would matter anyway. They’d be off maximizing paperclips and printing one-off ASICs for everything, or whatever

I guess by “good enough” what I actually mean is less so about them becoming super intelligent, and more about having established practices for getting LLMs to use modern best practices etc. For example, every thing they do is slightly out of date all of the time . So you get things like tidewave & context7, providing MCP tools to search docs etc, and tools like the one described here to gather rules and keep them up to date.

One clear hint that we are headed down this path would be if this incentive structure inverts.

That is to say: library maintainers begin to avoid changing things so as not to “break” models. Developer experience no longer matters because the developers are no longer developing, so the focus turns to not stepping on the models’ toes.

Why do I need Ash 7 when Ash 6 works fine and my cluster is churning out code? It’s not like my LLMs have an opinion about the changes!

Essentially, Python and JS become an intermediate representation between English and Applications.

I’m not saying this will happen, and it sounds like a dystopic nightmare to me! But there are shades of it already happening, like that paper.

The concept is predicated on the idea that all code with the same ultimate effect is of the same level of quality. There are so many axis to consider, and mostly LLMs have the same difficulties that human beings do (and a bunch we don’t). Too much complexity, too much spaghetti, makes it harder for LLMs and humans both to change applications over time and for those applications to be understandable.

I will place more weight on your experience here because I have not engaged in any LLM-assisted programming myself (I’m not convinced it’s useful at present, and I have privacy concerns).

But I am not convinced long-term that the failure modes of models and humans will be exactly the same, and I worry about excessive anthropomorphization here. Models can experience a codebase in a single forward pass, augmented by their training data. It could well end up being that abstractions which are useful to us humans (co-location of behavior, modularization, and so on) are less helpful to models. It could be that, if they get a bit smarter, they will be capable of maintaining that spaghetti - at least well enough to generate revenue.

After all, as we are all well aware, there are many companies making a great deal of money selling spaghetti even now

Similarly, a self-driving car has little use for a stop sign as it can see in every direction at once. Our heads are on a swivel.

That’s fair, and long term things might change, but by my estimation we are at an absolute minimum multiple years away from LLMs being able to work on code for a serious project absent supervision and human code reviews. So code still has to work for humans too.