LLMS & Elixir: Windfall or Deathblow?

Having spoken to Zach at ElixirConf EU and heard some of the challenges he has seen with supporting users that have completely fictional code. Coupled with his general skepticism of the AI hype. I am glad he wrote something up.

This is a pragmatic take on what we can usefully do around adoption, and being on a slimmer end of the bell curve when people start using bell-curve-machines for information and code generation.

Without trying to go to the moon or saying the cost of programming will go to zero. That stuff is speculative. This is mostly addressing current reality, whether you like the situation or not. I don’t love dealing with LLMs but I don’t particularly want to close my eyes because I don’t think it will entirely blow past.

I’ve only just started using AI and I have yet to use an agent. I’ve also never really chat as there is something something I find so off-putting out it. This article has some nice tips and some good insights that I hadn’t thought of. I particularly like the tip of asking it to first find the docs. This is something many people clearly take for granted.

Ultimately you should not be relying on an LLM as a source of “knowledge”.

I’ve seen several questions here, on reddit, and watching people IRL where they simply refuse to use anything other than a chatbot to figure things out. When they give up they don’t look to the docs, they ask someone. I know I’m not the only one wondering this but I haven’t seen too many satisfactory answers: what happens when people stop writing documentation? Are we going to start seeing libraries where the authors haven’t looked at a line of code? If people are producing software like this, how do the LLMs learn to use it? They’d have to start analyzing the code, of course, but how is that not going to degrade the slop even further?

Not trying to be a doomer just wondering, and wondering is all we can do.

Best case AFAIC is that we get flooded with so many quick-to-market apps that carefully crafted software with performance and security as first class citizens becomes cool again. And by “cool” I mean every day users will feel a noticeable difference.

I’ll tell you what my real experience is, using LLMs has allowed me to write more well tested code, as well as more well documented code. For example, we’ve been having discussions about some issues w/ Ash’s documentation.

1. the search results didn’t seem ordered by the most helpful stuff
2. there is so much content that “cross referencing” is important, i.e putting links to things you were probably looking for if you got here and its not what you want.

I spent about 2 hours w/ Claude working on our four most commonly used modules, Ash, Ash.Changeset, Ash.Query and Ash.ActionInput. We have a solid usage rules file and other things that helped guide its output. I tasked it with expanding the docs on the most common functions (and many of the less common ones) ensuring that we have example usage, cross links to other relevant guides, and typespecs. It wasn’t “vibe coding”, i.e I reviewed all of the work, made many tweaks of my own, etc. But it turned a multi-day project into a 2-3 hour project. And when we have other docs issues in the future I will likely feed it into Claude and have it generate some nicely formatted output. i.e “this user had trouble with X, please look at the current docs for Y and produce something that would have prevented this confusion”. It also figured out why the search results were suboptimal before I could figure it out myself. The order of modules in the sidebar controls to some degree the order of modules in the search. We reordered the sidebar, and search got that much better

If you’re familiar with the issues around Ash docs, the improvement to this module is effectively night and day: Ash — ash v3.29.3

To me, it’s not a replacement for my brain, it’s just a way for me to exert more leverage than I would have before. When you know how to use it, it can make you faster, help you automate the boring stuff so you can focus on the important stuff, and get you past the blank-page problem.

It’s nice to hear that putting documentation/instructions in context helps the models write Elixir (and Ash) code at a level closer to popular languages. I am still somewhat surprised/skeptical about that result because I would expect the amount of, say, Python content to dwarf Elixir to the degree that Python code would still come out much better. The fact that these context tricks work so well indicates to me that these models are generalizing very well across languages (a point made in the article).

Obviously as an Elixir Enjoyer I do hope it works out this way in the coming years.

Maybe others have had a different experience with the “modern web” and “modern apps”, but personally I am reminded of a point I saw about AI slop on social media not having made much of a dent because so much of social media was already human slop that it was hardly noticeable.

Similarly, so many webapps are b2b-enterprise-saas-slop that I’m not sure if anything is really going to change. But there is one thing to keep in mind: if most webapps are slop, then most of the training set is also slop, and the outputs of the models will remain slop. So spending time working on your skills and building better software is not a waste, as many on the hype side are claiming, since it should be quite viable in the short-medium term to “outrun” the models. At least I hope so, anyway.

Absolutely, and I’m all for its usage as you describe. My comment was missing some context—I joined reddit not too long ago so I’d stop posting low quality rants here (it’s more or less worked ). There’s a faction of people on reddit who basically call anyone claiming you still need to understand how to read and write code a gatekeeper. I’m wondering if that’s what Anthropic et al is really pushing for… sorta like how I don’t know how to write raw machine code but still call myself a programmer.

Re: Ash docs, I was familiar with it having problems but could not put my finger on it. Taking a quick look again already the sidebar so much less scary-looking! Topics look more consolidated too, for example, I don’t have to look at three different pages for different usages of the same concept. I haven’t worked on my Ash project in a week or so so extra looking forward to getting back to it now.

Yes, I didn’t want to dive into this because I didn’t want to write a novel-length post. But I mean generally it would be nice if it somehow triggered a trend of these things becoming important all around. Like “first/quick to market” could well become severely cheapened if people are inundated by even more garbage than they already are. Maybe we’ll start to see “Zero vibes” in marketing materials. A boy can dream

There were some arguing that this was already happening even before the slop-coding boom, e.g. this Linear blog post. It does seem like people are starting to get fed up with this stuff, and I hope they continue to vote with their wallets

Unfortunately the first thing you will discover (as I have) if you even try to build better apps is that nearly all of the standard tooling and best practices are woefully inadequate and need to be thrown out, which is what I have been slowly doing. And again, this flies in the face of the “2 week MVP” strategy because even learning how to build apps right is a multi-year process for which there are scant few resources to help you along. It really does hurt.

Incidentally I do see Ash as an approach intended to solve some of those problems, though I’m not entirely sure if they’re the same problems that I have. But more to the point: AI models as they exist are not capable of the level of creativity to create such a solution, so there is still hope yet. Antirez wrote an amusingly specific post about this the other day, which I thought drove the point home well.

Literally the best ever summary I’ve read on how an LLM empowers already powerful programmers. I had 100% the same experience; LLMs serve me as (1) much better search engines and (2) make long and tedious work that I know how to do from the beginning, that much faster. Makes me more creative. I don’t focus on minutiae as much as before.

This last part comes with a huge red warning though; every now and then do return to the minutiae and details, lest you forget your attention to details! But even that is made much better by the fact that I can in fact be creative and detailed about things I care about, and have everything else be accelerated a little bit (or a lot).

Gemini being described as a “smart duck” to chat with is both super funny and super true.

Great discussion! I wrote about this topic two years ago: On using LLMs like ChatGPT to get things done • OVERBRING Labs

I find that it’s a major windfall, iff you use LLMs correctly. And by that I mean: knowing enough of the topic at hand to spot inaccuracies, knowing how to have a discussion to explore a topic without getting caught up in “blind alleys” (where the LLM insists on e.g. giving you a solution about something you are only tangentially interested in), etc.

In the end, LLMs are a tool, though not “like any other”, since they can do things that would be considered sci-fi a few years ago. Still, knowing when and how to use a tool, matters.