I like these proposed syntax but agree that it is tricky to find a single, short word that coneys the meaning. I tend to prefer let and acc over the other suggestions, but I like for with(x=0), i <- stuff do or something crazy like for |x=0, y=0|, i <- stuff do myself.
I’m not put off by with being used elsewhere but unsure if it’s possible to have both. Other synonyms could be carrying, using, keeping, holding, rebinding…
Since the beginning of this thread I’ve had a strong aversion to the use of let and I realise now it’s because in other functional languages let is used to introduce a binding which can’t be rebound. When I read let I see a value which will never change so I get some cognitive dissonance seeing it used in this proposal.
Regarding the example
for let(sources),
sub <- subs,
sub_formatter = Path.join(sub, ".formatter.exs"),
File.exists?(sub_formatter) do
formatter_opts = eval_file_with_keyword_list(sub_formatter)
{formatter_opts_and_subs, sources} =
eval_deps_and_subdirectories(:in_memory, [sub], formatter_opts, sources)
{{sub, formatter_opts_and_subs}, sources}
end
As other’s have suggested using map_reduce does seem far clearer to me. When I see this I can understand that we’re using a particular type of for comprehension. It specifies how variables are bound, what they’re initialised to, what the folding function is and what the return shape is (I have a hard time associating all of this with let).
for map_reduce(sources),
sub <- subs,
sub_formatter = Path.join(sub, ".formatter.exs"),
File.exists?(sub_formatter) do
formatter_opts = eval_file_with_keyword_list(sub_formatter)
{formatter_opts_and_subs, sources} =
eval_deps_and_subdirectories(:in_memory, [sub], formatter_opts, sources)
{{sub, formatter_opts_and_subs}, sources}
end
It also opens the door for other functions
for group_by, x <- 1..10, y <- 1..10 do
case {x > 5, y > 5} do
{true, true} -> :top_right
{true, false} -> :bottom_right
{false, false} -> :bottom_left
{true, false} -> :top_left
end
end
(one advantage of having the function at the start vs the current keyword style is that you don’t need an associated value for the syntax to make sense i.e.
for x <- 1..10, y <- 1..10, group_by: ? do
I do still have some concerns about the rebinding being a bit mysterious but the examples are quite compelling so I can get behind just learning it as a new special form.
I think the proposal overall is fantastic and very well thought through!
let seems clear enough once you know what it does – I’m not a big fan of either init (too ambiguous) or let with parentheses (looks too much like an external function).
I wonder if we’re seeing the adoption of let as a common signal for scoped variable bindings in Elixir; after all it was recently introduced in heex templates recently (<.form let={f} for={@changeset} phx-change="validate" phx-submit="save">).
I am largely happy with let but I think bind might be better for this reason. It communicates that you are binding and (re)binding the variable to the loop. +1 from me.
I’m sorry if I’m repeating someone else’s argument. This post already had 66 messages when I first saw it and I haven’t been able to read through the whole thing yet. The general sentiment I got from a quick look is that most people are in support of the proposal and what’s being discussed at this point are syntactical and implementation details.
I wanted to share an observation about the original post that starts with a problem statement but then goes on to present a new syntax without making a strong case for how it solves the original problem.
Jose’s solution to the traversal problem in Elixir:
{sections, _acc} =
Enum.map_reduce(sections, {1, 1}, fn section, {section_counter, lesson_counter} ->
lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter
{lessons, lesson_counter} =
Enum.map_reduce(section["lessons"], lesson_counter, fn lesson, lesson_counter ->
{Map.put(lesson, "position", lesson_counter), lesson_counter + 1}
end)
section =
section
|> Map.put("lessons", lessons)
|> Map.put("position", section_counter)
{section, {section_counter + 1, lesson_counter}}
end)
The new solution the uses for let:
{sections, _acc} =
for let {section_counter, lesson_counter} = {1, 1}, section <- sections do
lesson_counter = if section["reset_lesson_position"], do: 1, else: lesson_counter
{lessons, lesson_counter} =
for let lesson_counter, lesson <- section["lessons"] do
{Map.put(lesson, "position", lesson_counter), lesson_counter + 1}
end
section =
section
|> Map.put("lessons", lessons)
|> Map.put("position", section_counter)
{section, {section_counter + 1, lesson_counter}}
end
They looks mostly the same: both have the same shape, the same nested “map-reduce” traversals. So all of the original observations Jose made about the first solution–lack of reassignment, lack of mutability–hold for the second one as well.
Ultimately, the question posed in the original post “Therefore, how can we move forward?” remains unanswered. To me it looks like there was a motivating example to showcase how the new proposal could by applied in practice, but by the time we got to the end of the proposal, the motivating example had been forgotten and turned out to be irrelevant to the discussion.
I don’t really get the rationale behind the additional syntax being more beginner-friendly. On the contrary, it seems to bring more dilution to the notion of “idiomatic Elixir” because different people will prefer using different constructs to do the same thing. For example,
iex> sum = 0
iex> count = 0
iex> for let {sum, count}, i <- [1, 2, 3] do
...> sum = sum + i
...> count = count + 1
...> {i * 2, {sum, count}}
...> end
can already be written as
Enum.map_reduce([1, 2, 3], {0, 0}, fn i, {sum, count} ->
sum = sum + i
count = count + 1
{i*2, {sum, count}}
end)
and the differences between the two are superficial.
So far, the interesting bits about the proposal are the possibilities Jose mentions towards the end of the original post, the ones that aren’t going to be supported initially. But the tradeoff of starting on the path towards those possibilities is making it more difficult to decide which way of writing any given piece of code is the preferred one, given that the number of alternatives keeps increasing.
Not meaning to derail the discussion but I just have a feeling that for’s self-containedness is what’s motivating the addition of new syntax. What if we could “break out” of its “do-end” box and start using it as part of idiomatic Elixir pipelines? This could be achieved by adding a way to get a stream out of for, say,
{combinations, {sum, count}} =
for(i <- [1,2,3], j <- [:a,:b,:c], as: :stream, do: {i, j})
|> Enum.map_reduce({0, 0}, fn {i, j}, {sum, count} ->
sum = sum + i
count = count + 1
[{i, j}, {sum, count}}
end)
Hi,
How about introducing something similar to LOOP macro in Lisp? With some properties like :collecting, :maximize, etc. It would give a complete set of options to be generic for most cases. If that would be reduce or map would depend on the annotations made by the user. I’m curious how difficult it would be to write LOOP macro in Elixir? In fact I always saw list comprehensions as a smaller brother of Lisp’s LOOP macro. But maybe that was only me…
P.
Yes, they look quite similar, but that’s partially the point. If we introduce something too different, then it is ultimately going to be rejected because it is not similar to anything in Elixir.
At the same time, if it is too similar, then people may say “well, it doesn’t add much”. And overall that’s a hard line to balance.
The motivating example brings two concerns:
The current solution has too much noise
The current solution requires knowing precisely the magic incantation to solve the problem (the word map_reduce)
If you go through the proposed guide and try to solve it with Enum, you are going to see that each step requires knowing a particular function. First you map, then you flat map, then flat map+map, then you filter, then you map reduce, etc. We have even discussed examples that have to use flat+map+reduce.
The whole point of for is that with the addition of two constructs (let and reduce), we can express almost everything in the Enum module. From any?, to find_value, to flat_map_reduce, to map_reduce, and to reduce, but without imposing all of this naming upfront. You have a single construct with one or two variations on top, and that’s it. Oh, and it also slightly reduces the amount of noise too!
This argument does not hold by itself because of this:
Well, we already have recursion, so why have the functions in Enum?
Well, we already have Enum, so why have for comprehension in the fist place?
Let’s say we go back in time and use this argument to not add for to the language. After all, for is a different construct for doing the same thing as Enum. Here is what would happen. We would have to rewrite this:
for {encoding1, value1} <- alphabet,
{encoding2, value2} <- alphabet do
encoding = bsl(encoding1, 8) + encoding2
value = bsl(value1, shift) + value2
[clause] = quote(do: (unquote(value) -> unquote(encoding)))
clause
end
to this:
Enum.flat_map(alphabet, fn {encoding1, value1} ->
Enum.map(alphabet, fn {encoding2, value2} ->
encoding = bsl(encoding1, 8) + encoding2
value = bsl(value1, shift) + value2
[clause] = quote(do: (unquote(value) -> unquote(encoding)))
clause
end)
end)
From this:
for %{pid: pid} <- files,
{_, _, ref, ^pid, on, _, _} <- waiting,
not defining?(on, waiting),
do: {ref, :not_found}
to this:
Enum.flat_map(files, fn %{pid: pid} ->
waiting
|> Enum.filter(fn {_, _, ref, waiting_pid, on, _, _} -> waiting_pid == pid end)
|> Enum.filter(fn {_, _, _, _, on, _, _} -> not defining?(on, waiting) end)
|> Enum.map(fn {_, _, ref, _, _, _, _} -> {ref, :not_found} end)
end)
From this:
for {pair, _, meta, _} <- all_defined,
{local, line, macro_dispatch?} <- out_neighbours(bag, {:local, pair}),
error = undefined_local_error(set, local, macro_dispatch?),
do: {build_meta(line, meta), local, error}
to this:
Enum.flat_map(all_defined, fn {pair, _, meta, _} ->
Enum.map(out_neighbours(bag, {:local, pair}), fn {local, line, macro_dispatch?} ->
error = undefined_local_error(set, local, macro_dispatch?)
{build_meta(line, meta), local, error}
end)
|> Enum.filter(fn {meta, local, error} -> error != nil end)
end)
From this:
for path <- Mix.Dep.load_paths(dep),
beam <- Path.wildcard(Path.join(path, "*.beam")),
Mix.Utils.last_modified(beam) > modified,
reduce: {modules, exports, new_exports} do
{modules, exports, new_exports} ->
...
{modules, exports, new_exports}
end
to this:
Enum.reduce(Mix.Dep.load_paths(dep), {modules, exports, new_exports}, fn path, acc ->
Path.wildcard(Path.join(path, "*.beam"))
|> Enum.filter(&Mix.Utils.last_modified(&1) > modified)
|> Enum.reduce(acc, fn beam, {modules, exports, new_exports} ->
...
{modules, exports, new_exports}
end)
end
I could go on and on. If we didn’t have for, all of those snippets would be worse, noisier, and also slower.
And I didn’t have to look hard either. Those examples came all from lib/elixir/lib, with the exception of the last one were I grepped for reduce: . I am sure you will find several others there, all in a single repository.
So my point is: yes, they ultimately achieve the same thing, but for comprehensions have always enabled us to write complex traversals quite more elegantly than the Enum module. If someone asked me what is Idiomatic Elixir, I would take for over Enum, whenever that’s possible. The same way I take Enum over manual recursion. This proposal allows us to use for in places where we can’t right now and that’s bound to bring the same benefits as we have seen in the snippets above. I have already posted examples of how the proposal can improve existing code.
Just wanted to throw my 0.02c onto the discussion. At first I didn’t like the proposed changes, but the more I’ve read this thread and let it sit with me the more it’s grown on me. I would like to express my support for two proposals others have made, namely I like bind over let, and I also think that it should require parenthesis. So to me, this is ideal in terms of readability of all the proposed options.
for bind({x, y} = {0, 0}), i <- [1, 2, 3] do
{i, {x + i, y + i + 1}}
end
Along with the required parentheses, an option to cover both map reduce and reduce without a separate word could be to add something like :returning. This could default to :map_reduce, but be set to :reduce as needed.
Map Reduce
iex> for let(sum = 0), i <- [1, 2, 3] do
...> sum = sum + i
...> {i * 2, sum}
...> end
{[2, 4, 6], 6}
Reduce
iex> for let(sum = 0, returning: :reduce), i <- [1, 2, 3] do
...> sum + i
...> end
6