Allowing an ignored member in list construction?

Chat & Discussions Proposals: Ideas
1

One common problem we face in constructing lists is that there is (AFAIK) no support for conditionally inserting members into list declarations.

You can’t write:

allow_even? = false
[
1,
if allow_even? do 2 end,
3
]

because the result would be [1, nil, 3], though syntactically a lot of code would be easier and more readable for us if you could do that.

What if returning a very specific value, like :ignore_list_member would lead to a list being constructed like this:

[1, :ignore_list_member, 3] ==> [1, 3]

Functions could return specifically this value if there’s nothing to insert.

def foo(x), do: if rem(x, 2) != 0 do x else :ignore_list_member end
[
foo(1),
foo(2),
foo(3)
]
===> [1, 3]

Right now we typically do something like piping a whole list like this:

allow_even? = false
[
1,
if allow_even? do 2 end,
3
]
|> Enum.reject(&is_nil/1)

which is fine, but I wonder if this could be solved more conveniently and generic without the need to walk the list a second time after constructing it - just once when it’s “constructed” with the declarative syntax.

If it’s not possible with small effort, I still thank you for reading this. :slightly_smiling_face:

2

Ages ago I did look into mimicing dart lists, which do have an add item conditionally synta as well as spreading loops onto the list:

2 Likes
3

I was thinking about this language feature too. As @LostKobrakai mentioned, this is a core feature in Dart. It’s documented here: Collections | Dart.

I came across a very nice podcast, where Robert Nystrom explains how this works in Dart, and why they choose to make it part of the language (tldr; because it’s handy to build collections like this, in a context where you’re building UI’s declaratively).

LV side-steps this problem with class lists by ignoring falsey values in these lists. I’m not sure the pull towards this feature is just as strong in Elixir, as it was in Dart. Afaik this can’t be emulated with macros, or something similar, but must be a core language feature.

I’m curious if there were previous discussions about this feature, and if there was consensus in the past about it.

1 Like
4

After I posted a macro implementing those features :smiley: It cannot be done without extra syntax, though elixir won’t be able to do so either. It only has a but more flexibility around new syntax over macros.

5

That’s an interesting proposal, but looking at the implementation it requires list concatenation underneath if I understand it right.

That would make it very similar to what we’re currently doing:

[1, 2, 3] ++
some_function_returning_a_list() ++ 
[4, 5]

which could also handle if constructs

[1, 2, 3] ++
if x do [:something] else [] end ++ 
[4, 5]

I was kind of hoping that the declarative syntax could be done in such a way at language level that it can avoid paying for concatenation in that way. (Unless ++ concatenation is very efficient. I always resorted to [ x | xs ] syntax where possible assuming ++ comes at a higher cost.)

6

Haha, I see where I messed up. I mean without wrapping the list as a whole :sweat_smile: I was thinking of a solution that doesn’t require a certain context to be set up. Maybe only a marcro inside a list.

7

Since this is about list literals, I wouldn’t worry too much about performance.

8

What is the problem of using macro proposed by @LostKobrakai ?

I’m not a fan of this becoming a language feature, as it involves a lot of mental overload vs not having to write 2 more lines of code.

9

If the value is conditionally there then there is no avoiding paying for concatenation. Lists are linked lists, they’re built from back to front. If part of the back is different depending on a runtime value then you have to pay a runtime cost to deal with it.

3 Likes
10

You can use the filter of a for comprehension

iex(9)> require Integer
Integer
iex(10)> allow_even? = false
false
iex(11)> for x <- 1..3, !Integer.is_even(x) || allow_even?, do: x
[1, 3]
iex(12)> allow_even? = true
true
iex(13)> for x <- 1..3, !Integer.is_even(x) || allow_even?, do: x
[1, 2, 3]

or you could use Enum.flat_map but then you’re creating intermediate lists

iex(14)> Enum.flat_map(1..3, fn
...(14)>   x when not Integer.is_even(x) or allow_even? -> [x]
...(14)>   _ -> []
...(14)> end)
[1, 2, 3]
iex(15)> allow_even? = false
false
iex(16)> Enum.flat_map(1..3, fn
...(16)>   x when not Integer.is_even(x) or allow_even? -> [x]
...(16)>   _ -> []
...(16)> end)
[1, 3]
3 Likes
11

On the very rare occasion I’ve done this I just used a list of lists (where the conditional adding of a thing added a non-empty list, and if the condition was false then it simply added an empty list) and then just do List.flatten at the end.

I understand the appeal of having such a language construct but it’s IMO too niche.

4 Likes
12

I think you’re missing the point, @gregvaughn, that I’m trying to make as I was using an arbitrary example instead of constructing a real list that cannot be rewritten by a mere list comprehension.

The point is to construct lists with several different optional elements, each with its own condition (which occurs in my code base). Right now we either reject nil elements afterwards or flatten the overall list. It’s easy if everyone agrees how to handle optional elements.

My real example is data used describing steps in a protocol tester, which would look more like this:

[ < ... data representing step 1 ...>,
  < ... data representing step 2 ...>,
  < ... data represening optional step 3 ...>,
  < ... data representing step 4 ...>,
  < ... data represening optional step 5 ...> |
  < ... data representing further test steps ...> ]
  |> Enum.reject(&is_nil/1)

Each of these data structures is generated by calling a function to make the respective step.

If there was a way to construct such lists without having to filter or flatten them, it would be a boon in several places, but none of them really offer themselves for a list comprehension unless I missed your point.

13

Hello, @dimitarvp. Yes, that’s what I’m currently resorting to in some places.

Fair enough if it’s too niche!

14

I already have a solution that works with one more line of code, I just thought I encounter the problem so often in making data structures, other people must have it too. Seems like not.

My software really builds a lot around big, declarative lists with structs and further nesting for the protocol data and also for the FW data. I encounter this particular thing a lot describing my test data.

15

I suppose I did miss the point. I haven’t experienced the pain/annoyance you’re having. This sounds like a specialized technique and/or something you’re familiar with from prior languages/frameworks.

One more thought crossed my mind. Could this be an XY problem? Do you really need a list, or would an enumerable suffice? If an enumerable is good, then I might create a stream that encapsulates this skipping logic.

2 Likes
16

Maybe I’m using the wrong term here.

Let’s say one element in the middle of the list was optional. Still it would be possible to walk the construction of the list only once at runtime, I thought, irrespective of order? Like only paying it once.

Let’s say I’m building a list declaratively from functions each returning one arbitrarily complex item:

[ foo1(), foo2(), foo3() ]

What is the cost for constructing this list? O(1)? O(n)? Is it paid at compile or runtime?

And if I do this:

[ elem1, optional_elem2, elem3 ] |> Enum.reject(&is_nil/1)

Is the cost then O(n) * 2? (Do I walk the whole list twice?)

I guess that’s what it all depends on - because my assumptions of what the costs of these constructs are are wrong, then I might understand the value of various solutions better.

1 Like
17

@gregvaughn - sorry for the misleading example then. :slightly_smiling_face:

It doesn’t have to be a list, you’re right about that. I’m merely interested in two things:

  1. Preserving the order.
  2. Being able to write it in a declarative way, item after item.

If you have a good alternate proposal, I would be delighted. :slight_smile:

18

At a minimum, I think this would need some alternative syntax because this is already valid Elixir:

[
  :foo,
  if x.something? do :bar end,
  :baz
]

This is distinctly different from Dart, where the regular if is a statement with no value so putting it inside an expression would otherwise be meaningless.

That syntax would also cause headaches for referential transparency, since this should be “the same” code as the above:

extra_var = if x.something? do :bar end

[
  :foo,
  extra_var,
  :baz
]

IMO the better approach is to use a domain-appropriate “empty” element and clean it up if needed. Cleanup may not even be strictly necessary - for instance, if you’re producing a big iolist value with optional parts then [] is a perfectly fine “skip this and go to the next element” marker.

5 Likes
19

Yes, but this happens rather a lot and is very well optimized. Eg an Enum.map is a 2x walk because you have to build up the result backwards and then reverse it (or use a body recursive stack, but that’s still functionally a 2x walk since you have to pop the stack).

If you have literally just one optional item btw you can:

if optional_thing do
  [a, optional_thing, b]
else
  [a, b]
end

If you are trying to construct a list that has many different items that may optional interspersed amongst items that are not, I don’t see how the compiler is supposed to do constant optimization. Definitionally you aren’t constructing a constant, you’re constructing a value who’s length varies by a bunch of different values.

Overall I think this discussion of costs is probably misplaced. Are we talking about lists that are literally a handful of items long? If so we’re in hyper optimization land and the code that performs best will rely heavily on exactly what you’re trying to do.

6 Likes
20

I strongly second this, it probably shouldn’t matter in practice or if there was a real performance concern we would need to benchmark to find an optimal solution.

Just code golfing at this point for the sake of the argument (and fun), I came up with the following which should do it in one pass (well two technically since the list gets built backwards and reversed):

defmodule POC do
  @steps [&POC.foo/1, &POC.bar/1]
  
  def foo(_ctx), do: nil
  def bar(ctx), do: ctx.bar
  
  def only_truthy_results(ctx) do
    for step <- @steps, res = step.(ctx), do: res
  end
end

The trick being that since @steps is a literal here, the input list does not need to be built on the fly, it just has to be moved in memory. And the comprehension will directly build the filtered list. You can pass arguments in the form of a context map if you need to.

1 Like