I talked chat gpt through writing me a neural network with a exponential decayed potentiation to increase the likely hood of re triggering again after a recent trigger event in elixir. It was on par with a junior ish developer in terms of what mistakes it would make but it was fast and far cheaper than my hiring someone to basic plumbing/setup work. It will be a mistake to overlook this area regardless of the immaturity of the current generation of tooling.
noizu
ajur58
I was pumped by ChatGPT at first, now I’m just worried of the damage it will do. I’ve used it almost daily in the past month, for ideation, code generation (learning Elixir), asking quick facts, and it has a scary high ratio of just giving you garbage super confidently, both with code and simple conversations.
For example, I asked when a certain person won the world championship in some sport. It gave me a wrong answer. But it was so wrong, that it even gave completely fictional details about city & country of contest, athlete names, years. When pressing that the answer is not correct, it made up stuff again. This went on for 5 or 6 rounds, confidently spitting out lies. When this happens with code, especially if it’s generating issues with edge-cases that you don’t realize at first, there’ll be way more time lost than won with the scaffolding.
I’ll park my initial enthusiasm and will revisit ChatGPT & co in a couple of months.
3
josevalim
Creator of Elixir
I believe this was written in a confusing way. The following chart from the article is a bit clearer in that those numbers mean suggestions acceptance rate. So 46% of the suggestions were accepted but this is a quantitative measure and it makes sense they are higher for Java as it has more boilerplate. But here is what I would love to know:
-
If 54% of the suggestions are rejected, does it mean I need to parse a suggestion and then discard it? Which would mean that most of the time suggestions could be slowing me down?
-
What is the time taken to accept or reject a suggestion?
-
Does Copilot tracks what happens with a suggestion? Maybe it is accepted and then it is immediately changed or removed because it was wrong?
In any case, I believe there are two separate discussions here, and they are getting mixed.
-
Are the AI tools in a state where we can consider them trustworthy or generally acceptable? To me the answer is no. Besides a huge potential copyright issue on tools like Copilot, which has made some organizations ban certain AI tools altogether, there is still a lot to improve. For example, researchers have found that code generated by OpenAI’s Codex contained security vulnerabilities 40% of the time. However, the tools will improve as there is a large amount of techniques and ideas that still have to explored and potentially adopted (such as reinforcement learning with static and security analysis!).
-
That brings us to the second discussion: should we do what is necessary to get Elixir working with more AI tools? To me, the answer is a 100% yes, because it will only get better and it is not only about code completion.
15
Sorc96
I think we’ve seen this here on the forum as well. Newbies pasting a piece of code and asking why it doesn’t work, without explaining what they were trying to do with it in the first place. Often, the code isn’t even syntactically valid.
Also, I remember reading about somebody asking how to transpose a matrix in Elixir. ChatGPT insisted that there is a transpose function in the standard library. It even generated a working implementation when asked to show the source code.
4
tfwright
My last comment on ChatGPT will be this:
I asked a dev on my team to build a pretty simple feature using Phoenix channels. We sketched out the basic architecture, they said they understood, and off they went. Some very long time later they came back with a working prototype, but it did not adhere to our design. Specifically, it was pushing data from one channel to the client, which was sending it back up to a different channel, rather than sending it from one process to another directly as we planned. When I asked what was up with that, they said they couldn’t figure out how to get that piece working, and they asked ChatGPT which told them it was not possible. So there was a whole suite of javascript functions responsible only for listening for different data events and sending them right back. This was despite having the relevant documentation at hand. But why fuss with sometimes confusing documentation when you can just ask an elaborate magic 8 ball?
My first experience with “AI” programming in the wild and sadly, yes, probably not my last.
6
BartOtten
I can’t wait to write “Write me a state machine in Elixir which can keep track of daily routines” and then “Write me a LiveView frontend for the backend in the visual style of other views” and knowing the code is excellent.
This way I would only have to create really unique things that require human capabilities. No more plumbing, just “snippets on steroids”.
But we are not there yet.
Ps. We better learn how to “ask” best as the quality of the input largely influences the quality of the outcome. For reference: geek vs noob search results 
1
zachallaun
I haven’t used this to generate code, but I’ve asked a few Elixir-related questions using https://phind.com/ and have had great results. The responses all have references, so I’ve found it to be a pretty great “shortcut to the right docs” in a couple cases where it wasn’t clear where to go first.
5
D4no0
After reading the posts, it seems that there are mixed opinions, the only question that was not addressed: can a matrix of computed weights (witch such technologies are) that is not capable of understanding concepts be capable of solving a problem? the short answer is no.
If such a technology is capable of replacing people, then it means that the work they do could be automated long ago, AI was never needed for this, all this scaffolding and plumbing problem you are talking about so often seems to be more of an organizational problem than related to the limits of engineering.
As for my last five cents, products like copilot and chatgpt will always be biased, because they are commercial products and their main scope is to make profit, not your life easier, this is true now more than ever, where consumerism is more important than innovation.
3
Cochonours
That’s crazy. I understand using ChatGPT to test its capabilities and play with them, but there are actually developers using it to help them code?
IMHO most people in this thread underestimate how much AI will disrupt our space, just like artists and designers thought for the longest time that AI would never be a problem for them. Nothing to really worry about for the next few years, but I dread a little being forced to use this kind of tool in order to stay competitive in the future.
2
tfwright
I think the point is developers can use AI to help them not code. That’s the selling point. It sounds good, because people hear it and think “more productivity, less work”, which is not untrue. Of course, like any tool, there are dangers to being too reliant on it. But in the case of AI, I think the very words we use to describe it, starting with “intelligence,” give the false impression that if you have this tool, you don’t need to know what you’re doing. And that is something we should all be very suspicious of, concerned about even, and not really because we fear for our jobs.
2