Elixir’s AST isn’t specified so they were relying on implementation details anyway.
dimitarvp
tmbb
The AST is definitely described in the docs: Syntax reference — Elixir v1.14.5. As such, it’s very common to rely on the exact format of the AST (most macros can’t be written if the concrete form of the AST can’t be assumed to be of a given shape).
1
dimitarvp
Yes, but as far as I am aware there is no promise it will never change. At least I remember that said in the forum a few years ago. Things might have changed.
0
hst337
I am strongly against this proposal, I have objective points and I’ll explain them one by one
- The problem is not important.
- Existing solution is okay.
- There are other solutions for other traversals which do not require any changes in the language.
- Mutable variables bring more confusion.
- Proposal is unclear.
The Problem
Nested structures traversal has always been a difficult problem for Elixir, because language is immutable (therefore nested traversals require careful rewriting of the parts of the structures) and Access is quite limited, since it is not working with structs out of box, it is missing most common lenses for different traversals and it is just not popular (most of the Elixir developers I’ve worked with didn’t know about `Access.key
Described solution
It is okay
-
Amount of code. Yes, it doesn’t look as short as Python’s solution, but it has lower cognitive load (it can be read from top to bottom, without keeping in mind mutable variables and when can they be changed).
-
“Lack of reassignment” and “Lack of mutability” can be downsides only for developers coming from the other languages. It is a great goal to have lower learning curve than other languages, especially for functional language aiming at Ruby and Python developer’s auditory, but I’ll describe why local mutable variables will make everything more difficult in the separate point. This point just states that for Elixir developers, these are not downsides.
-
It is not hard to write. When I did Elixir teaching,
Enum.reducewas taught on 2nd or 3rd lesson, and students were using it comfortable after a first week or so. These kinds of problems are solved within a 5 minutes max. -
It is not hard to read. And this point comes from immutability. In immutable languages, developer doesn’t need to track every change of the variable.
Other solutions
However, there are third-party solutions for these kinds of problems which work this just fine. I don’t want to rename all of them, but I know for sure that they exist.
Local mutable variables will bring more confusion
And this is the most important point. I want to make clear that I am explaining this not from a point that always immutable variables are better, but from a point that exactly for the language in it’s current state, it’s current design and it’s current ecosystem, they will do more harm than good.
Context-dependency
-
Technical differences will require rewrite, while semantics will stay the same. It is clear, that mutable variables will be used as a part of
formacro and possible other macros/functions for enumerables traversals. However, the exact context where this feature will be applicable is unclear. For example, let’s take pretty regular optimization, where some traversal is parallelized withTask.async_stream. I am pretty sure that it is not a use case for local mutable variables, however it is a list traversal semantically. Or, as far as I can see right now, it will be applicable tofor, but not toEnumfunctions. These are different versions of the exactly same approaches to the problem, but with local mutable variables they’ll have to be read and written in extremely different ways. -
Unclear edge-cases. Nested
fors? Twofors in a row?forwith a macro over avaluewhich doesn’t know about this mutability feature. If I use aforand define a new variable like
mut value = 0
for entry <- list do
[
value = value + entry,
value = value + entry + 1
]
end
What will it return where list = [123]? [[123, 124]] and value = 124? Or [[123, 247]] and value = 247? Extremely unclear.
It will make learning process harder
Having a single functional approach with high-order traversal functions is a feature of the language. Small core of the features, almost no intrinsics are what Elixir is loved for. Having a special context-dependent language feature, which semantically goes against the general language design and is an exception, is always a hard to learn thing.
180 degrees opposite to the mutable languages in terms of performance
for macro is already underperforming. Plus, both Elixir and Erlang are missing loop optimizations (and it seems that I am the only one who cares about it). Loop written in a Python style will generate performance penalties. For example,
def sum(list) do
mut value = 0
for entry <- list do
value = value + entry
end
value
end
Will be underperforming, since it’ll have to generate a list and return a value, but semantically it is clear that list is not required here. And Python-to-Elixir developer will have an incorrect feeling that this is the fastest way to sum a list, while it is actually a slowest. Any compiler with top-level optimizations would be able to optimize it, Elixir’s one is uncapable of doing it.
for is already not popular
I’ve just grepped the opensource I have locally cloned. for is used 10x less frequently than all of the Enum functions (and most of the time it is used in .heex templates, because it requires less escapes). And I can tell that it is for a reason. for has lower performance, it has too many features and it is really easy to misuse because of it’s filtering-by-pattern approach. Adding a new feature won’t make it more popular, it will make it more feature-creeping.
Proposal is unclear
After writing this text it is still unclear for me what problem is actually being solved by addition of local mutable variables. We’re already able to efficiently traverse nested structures and we’re able to do it from “readable but slow” to “the most efficient” way.
Perhaps new feature improves readability? Perhaps this feature will make it easier to switch languages from Ruby to Elixir, for example? Perhaps this feature will make our code more efficient? Perhaps this feature will solve some other problem? I’ve read the subject post carefully several times, and it is still unclear.
0
josevalim
Creator of Elixir
Hi @hissssst, I will focus my reply mostly on the inaccuracies and disagreements (and I will reopen the thread so the discussion can continue).
First of all, I understand some believe that the problem is not worth solving and the current approaches are fine. I don’t think our opinions will change on this one, so I will skip this part.
I would love to see a solution to this problem that uses Access and traversals as a starting point. It is a route that I have explored but I could not come up with anything satisfactory. So if improving Access is answer, I would like to see something concrete towards this direction.
Please name them, otherwise it is hard to agree that there are solutions to this problem if we can’t see nor evaluate them.
It is not unclear. Local mutable variables do not cross functions (anonymous or not), as they would no longer be local. This is mentioned in the proposal and in the discussion. You are correct that using them would require reframing the problem, but so would any Enum.map_reduce (the current solution) that needs to become a Task.async_stream.
Please provide examples, as I fail to see how these can be edge cases. The only edge case you gave as an example it is not unclear nor new. Let’s take a look at it:
mut value = 0
for entry <- list do
[
value = value + entry,
value = value + entry + 1
]
end
To answer what is the value of value, it is the same as if you wrote this:
value = 0
entry = 123
[
value = value + entry,
value = value + entry + 1
]
# value?
There is nothing new to the language semantics here.
This is 100% inaccurate. Today Elixir already does not generate the list of a comprehension if the result is unused and it would be trivial to extend that to this proposal.
No, you cannot tell us it is for that reason, unless you do studies with the community that proves that’s the root cause. Enums also provide more functionality that cannot be expressed with for and one for can be a replacement to several Enum calls, so this measurement is hardly a concrete measure of anything.
You said earlier that not many people are interested in improving comprehensions. If their performance was a concern, as you claim to be, wouldn’t we see more people interested in speeding them up? So these two arguments seem to contradict each other. In practice I haven’t seen anyone, except you, concerned about their performance to the point where their usage would be a deal breaker, and we are always glad to continue improving things where we can.
Look, I really struggle joining discussions where opinions are presented as facts. They are very draining because I have to be the one presenting logical arguments to things that are often subjective. It is ok to say your opinions but it is unfair to present them as something other than that. I like to think I have a pretty good feeling of the community needs and asks, and even then I try to be careful to not present my perception as a single and correct point of view, and I would like to ask you to do the same.
11
hst337
Yes, sure, I just didn’t want it to sound like advertising. One is Pathex traversal with pdict, other is Iteraptor by @mudasobwa, which is more suitable for this exact problem.
Than it is not mutable in a traditional sense, it is semi-mutable. Erlang lists compilation works in a way, that context is not inherited between instantiation of list items. But “mutability”-as-a-term means that it is. That’s the problem which can not be easily fixed on the level of Elixir’s compiler.
To rephrase it, Python (and any other mutable language) dev would expect
x = 0
[
x = x + 100500,
x = x + 1
]
To be [100500, 100501] and x = 100501, but in Elixir this would actually be [100500, 1] and x = 1. This is not mutability by definition.
You can say that you didn’t mean classic mutability, but I can say that it will be a huge problem to learn how this kind of semi-mutable and semi-immutable approach works for both functional and oo devs.
That’s true, I didn’t mean that my opinions were facts. I am not tagging anything as a fact or opinion, because I thought that reader will distinguish it on he*'s own. If you feel that something is not a fact (and my words about reasons of unpopularity of for were completely subjective), than it is not a fact and it is an opinion 
Considering the unpopularity of for, it is just my feeling and my experience from talking with other people. Most of them say “that’s how I was taught”.
What do you mean? As far as I know, Elixir’s for is just a wrapper around Enum. If I write
def send_messages(pids, message) do
for pid <- pids, send(pid, message)
end
It will generate
Enum.reduce(pids, [], fn pid, acc -> [send(pid, message) | acc]) |> Enum.reverse()
Which will instantiate a list of messages.
Vanilla compiler is uncapable of performing dead code analysis of inter-module call or which requires some inlining to perform correctly, since it requires analysis of whole graph of modules, which can change in runtime, and that’s what I meant. I can extend the sum example to reflect my point better, if you want
I didn’t say that. I meant that there is already a solution to this problem, and everybody knows how to use Enum to write this solution. Everything can be improved, even comprehensions. And I am working on it in my free time
That’s not quite true, but what you’re saying is technically correct. I’ve never seen any person or company choosing other language over Elixir because list comprehensions were slow (and that’s where I agree with you here). But I’ve seen a lot of companies switching from Elixir to Go, or from Elixir to Kotlin or to the other languages because of Elixir’s performance.
And if we’re talking about performance in general, loop optimizations are a part of them. And they’re very important, since first it is usually a low-hanging fruit and second they’re one of the few language features where runtime penalty heavily affects the performance (alongside with GC and dynamic typing dispatch).
So that’s why performance of Enum and comprehensions is important, and I once again want to mention @sabiwara for putting some effort into improving performance of Elixir library and Enumancer which became an inspiration for my currently WIP Enum pipes optimizer.
I’ve been exploring it some time ago, let me get my ideas together and I’ll present a solution within the Pathex update (and what can be done in Pathex, can be a part of Access for sure)
1
josevalim
Creator of Elixir
It is a difference in communication style then, thank you for clarifying. I also apologize for assuming you were passing your opinions as facts.
If you find it useful, I suggest making it clearer in your texts when something is based on something objective, such as “for is slow because of benchmarks which I can show”, and when it is an opinion. It can be as simple as adding “I believe”, “I think”, when you emit an opinion or something based on subjective experience, and providing evidence in concrete cases. This way you don’t leave it up to the reader to guess which is which (which would be a shame, because someone may also misjudge something objective you say as subjective). It is also important to make sure we are not just rehashing subjective opinions in discussions.
It is also always worth adding an extra effort in making your texts objective, if you can, as it leads to more productive discussions.
In your earlier example, it was clear within the local context that the result of for is not used, which is always the case when using local accumulators, as you need to read the value of the local accumulator if you want to use it afterwards. So you are right, we do not perform inter-module analysis but local analysis is enough here (or when you use comprehensions for purely side-effects).
Could the problem statement be solved by any of those? I am looking forward to other approaches you come up with!
0
adw632
This is just my opinion but I associate @ with module variables. Module variable behaviour is also different to assignment to an @@variable which does not collect/accumulate, it just binds a value.
Can we consider binding to outer scope as a concept, rather than decorating a variable as “mutable”.
For example adopt a special assignment/binding operator such as := for binding to the outer scope within the same function. Again this should not cross function boundaries, so anonymous functions cannot bind outside their scope.
The code would look like:
section_counter = 0
lesson_counter = 0
for section <- sections do
if section["reset_lesson_position"] do
lesson_counter := 0
end
section_counter := section_counter + 1
lessons =
for lesson <- section["lessons"] do
lesson_counter := lesson_counter + 1
Map.put(lesson, "position", lesson_counter)
end
section
|> Map.put("lessons", lessons)
|> Map.put("position", section_counter)
end
This could also work for things like:
{foo, bar} := some_func(args)
{:ok, bar} := another_func(args)
Anyway it’s just another idea on avoiding the mutability concept but providing the usability to bind to parent scope within the same function.
1
josevalim
Creator of Elixir
Unfortunately I don’t think the operator is good enough, given the different ways we can assign/bind to a variable in Elixir. And the fact that in the same pattern we can have both variables and local accumulators.
1
josevalim
Creator of Elixir
The new proposal is here: Local accumulators for cleaner comprehensions.
Click here if you want to jump directly to the new part of the proposal.
Everyone is welcome to post your opinions and feedback on the new thread, even if the new proposal does not really change the opinion you posted here.
I will close this thread. @hst337 (and anyone else), alternative answers to the problem statement with Pathex, Iteraptor, or anything else is very welcome in the new thread.
4