Is anyone using AI assistance in coding Elixir?

LOL I am stealing your quote!

I’ve been using Cursor AI for the last six or eight weeks. Been very helpful and a lot of fun!

An LLM can definitely lead you down a rabbit hole. I found that when I go back to good solid techniques, working in small steps with tests, It really, really shines.

I’ve been lurking on this discussion and it’s all very interesting.

Does anyone have a link to a video of someone live coding proficiently with a coding assistant? I still haven’t used one myself and whenever I see someone using one (be it in person or online) it always kind of looks like hell, lol. Code is flying in their face that they constantly have to stop and read to see if it’s right. Sometimes they accidentally accept it then have to undo… in fact, it really seems like there is a lot of rejecting the wrong thing be it parts of the suggestion of the whole thing. It’s neat when it can write a function for you, though I fall into the camp of this taking away one of the most fun things about programming—I enjoy the mental exercise of figuring out a solution on my own. For boilerplate I still use snippets, and for simple refactoring tasks it seems that so long as you have a certain level of editor mastery I’m not sure there is any improvement here (and again, this is another fun thing about programming for me).

I can absolutely see value in having it write a function for you when have no idea what it is you’re doing. Having it right in editor would be nice, though whenever I’ve used ChatGTP I still find myself looking up other solutions to compare results, thus I’d be switching to my browser anyway.

I may just be holding it wrong, though, and would be great to see someone using it really proficiently.

I don’t really “use” it so much deliberately but macvim/vimr has a GitHub copilot thing going and I haven’t disabled that. While I experienced quite a bit of hallucinating (eg code that looks good but doesn’t work in the slightest) in Python, the Elixir ideas copilot come up with are much more usable.
I don’t know if that is because I tend to code cleaner in Elixir or what else might be the reason.

I use chatgpt 4o canvas for targeted small tasks and high level architect brainstorming. I don’t use LLM for in-editor code completions. I have done enough prompt engineering and DNN models to see LLM as a glorified sentence transformer.

For posterity, I’m sharing how I structure my prompt so it may help everyone. I simply start a chat with sentence “Use your knowledge on elixir [programming language and coding patterns] to advise me.” where the bracket part is optional. Follow with a short blurb about the code I’m about to dump, then dump all the relevant code, finally write my request at the end, like “complete the TODO in the code”.

Here is the chat for that “check to see if two lists have the same members” task. ChatGPT . Since I asked it to advise me, LLM tends to provide options, and I believe the consensus in this thread is to use Enum.frequencies due to “the overhead of sorting”, that’s option 2.

I’m using copilot and it’s kinda convenient for PR reviews, on rewriting something simple to the standard you want and sometimes its suggestions are kinda what you really need. But sometimes not)

This is incorrect - Enum.sort(a) == Enum.sort(b) is faster than Enum.frequencies(a) == Enum.frequencies(b). At least that’s true for lists of numbers, it seems.

This is a bit messy, but here’s a bunch of varieties: benchmarking some list comparisons · GitHub

My output looks like this (m2 macbook air 8gb):

Operating System: macOS
CPU Information: Apple M2
Number of Available Cores: 8
Available memory: 8 GB
Elixir 1.17.3
Erlang 27.1
JIT enabled: true

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 2 min 55 s

Benchmarking freq diff length ...
Benchmarking freq different ...
Benchmarking freq list_reversed ...
Benchmarking freq list_sorted ...
Benchmarking freq same ...
Benchmarking sort by diff length ...
Benchmarking sort by different ...
Benchmarking sort by list_reversed ...
Benchmarking sort by list_sorted ...
Benchmarking sort by same ...
Benchmarking sort desc diff length ...
Benchmarking sort desc different ...
Benchmarking sort desc list_reversed ...
Benchmarking sort desc list_sorted ...
Benchmarking sort desc same ...
Benchmarking sort diff length ...
Benchmarking sort different ...
Benchmarking sort list_reversed ...
Benchmarking sort list_sorted ...
Benchmarking sort reduce while diff length ...
Benchmarking sort reduce while different ...
Benchmarking sort reduce while list_reversed ...
Benchmarking sort reduce while list_sorted ...
Benchmarking sort reduce while same ...
Benchmarking sort same ...
Calculating statistics...
Formatting results...

Name                                      ips        average  deviation         median         99th %
sort list_reversed                    97.38 K       10.27 μs   ±241.33%        9.25 μs       25.29 μs
sort desc list_reversed               68.78 K       14.54 μs    ±49.31%       13.25 μs          25 μs
sort list_sorted                      66.85 K       14.96 μs    ±32.73%       13.83 μs       25.29 μs
sort desc list_sorted                 50.79 K       19.69 μs    ±23.54%       17.71 μs       33.17 μs
sort reduce while list_reversed       46.92 K       21.31 μs    ±31.00%       18.21 μs       37.96 μs
sort reduce while list_sorted         38.17 K       26.20 μs    ±25.56%          23 μs          41 μs
sort desc same                        14.45 K       69.20 μs    ±13.73%       67.17 μs      100.51 μs
sort same                             14.08 K       71.03 μs    ±12.33%       69.00 μs      110.38 μs
sort diff length                      13.54 K       73.85 μs    ±12.99%       70.54 μs      109.81 μs
sort desc different                   13.33 K       75.03 μs    ±13.30%       72.08 μs      100.96 μs
sort desc diff length                 13.08 K       76.42 μs    ±48.78%       72.63 μs      123.55 μs
sort different                        12.97 K       77.11 μs    ±15.20%          74 μs      117.58 μs
sort reduce while same                12.91 K       77.47 μs    ±20.59%       73.04 μs      140.75 μs
sort reduce while different           12.54 K       79.77 μs    ±20.90%       74.92 μs      142.12 μs
sort reduce while diff length         11.37 K       87.94 μs    ±20.62%       84.50 μs      156.51 μs
sort by list_reversed                 11.10 K       90.07 μs    ±14.71%       88.25 μs      138.04 μs
sort by list_sorted                    9.96 K      100.44 μs     ±3.35%      100.13 μs      113.71 μs
sort by same                           5.29 K      189.03 μs    ±17.91%      184.25 μs      299.25 μs
sort by diff length                    4.95 K      202.01 μs    ±14.98%      191.29 μs      304.10 μs
sort by different                      4.78 K      209.37 μs    ±14.98%      204.42 μs      317.11 μs
freq different                         4.48 K      223.43 μs    ±14.48%      227.25 μs      290.60 μs
freq same                              4.39 K      227.63 μs    ±14.08%      229.25 μs      288.85 μs
freq diff length                       4.35 K      229.71 μs    ±16.38%      230.17 μs      301.77 μs
freq list_sorted                       2.92 K      342.31 μs    ±11.63%      337.21 μs      426.22 μs
freq list_reversed                     2.83 K      352.81 μs    ±11.30%      359.88 μs      431.96 μs

Comparison: 
sort list_reversed                    97.38 K
sort desc list_reversed               68.78 K - 1.42x slower +4.27 μs
sort list_sorted                      66.85 K - 1.46x slower +4.69 μs
sort desc list_sorted                 50.79 K - 1.92x slower +9.42 μs
sort reduce while list_reversed       46.92 K - 2.08x slower +11.05 μs
sort reduce while list_sorted         38.17 K - 2.55x slower +15.93 μs
sort desc same                        14.45 K - 6.74x slower +58.93 μs
sort same                             14.08 K - 6.92x slower +60.76 μs
sort diff length                      13.54 K - 7.19x slower +63.58 μs
sort desc different                   13.33 K - 7.31x slower +64.76 μs
sort desc diff length                 13.08 K - 7.44x slower +66.15 μs
sort different                        12.97 K - 7.51x slower +66.84 μs
sort reduce while same                12.91 K - 7.54x slower +67.20 μs
sort reduce while different           12.54 K - 7.77x slower +69.50 μs
sort reduce while diff length         11.37 K - 8.56x slower +77.67 μs
sort by list_reversed                 11.10 K - 8.77x slower +79.80 μs
sort by list_sorted                    9.96 K - 9.78x slower +90.17 μs
sort by same                           5.29 K - 18.41x slower +178.77 μs
sort by diff length                    4.95 K - 19.67x slower +191.74 μs
sort by different                      4.78 K - 20.39x slower +199.10 μs
freq different                         4.48 K - 21.76x slower +213.16 μs
freq same                              4.39 K - 22.17x slower +217.36 μs
freq diff length                       4.35 K - 22.37x slower +219.44 μs
freq list_sorted                       2.92 K - 33.33x slower +332.04 μs
freq list_reversed                     2.83 K - 34.36x slower +342.54 μs

the specifics aren’t too important, but it’s pretty clear that Enum.frequencies/1 is the slowest option. This isn’t an excellent test, and you should always profile. The point is, ChatGPT does not (and I would argue cannot) know the answer, so anything it spits out needs to be assumed to be false.

Unfortunately, a broken clock is correct twice a day. The same is, of course, true for LLMs, but very few people know if they can read a clock (to continue the analogy).

I am using Cursor and it is pretty epic. Works best if there are good examples in the codebase. 100% recommend.

+1 for cursor and its autocomplete on steroids

For the benefit of the community I would strongly advise you all not to use ChatGPT with Elixir. Their models simply do not understand the language. I burnt through easily 10k tokens today getting nowhere on a medium difficulty problem. Out of frustration I reach for the Claude free tier, and set it to “concise.” It solved the problem in 1k tokens “no sweat”! Before today I believed that the “frontier” LLMs were all roughly comparable when writing Elixir, but it’s clear to me no there are order of magnitude differences in capabilities.

Use Claude!