Elixir Enum module usage

So I finally hopped on Slack (it really is like a worse gitter, not even code syntax highlighting, but that is for another post), and someone there said, and I quote:

Afaik people avoid Enum anyway

I rarely ever touch Enum

Pretty sure @david.antaramian also said at one point he rolls his own rather than Enum (although I might be wrong)

I think the general consensus is Enum until you know how to write Elixir comfortably, then roll your own as needed

because iteration is rarely ever generic

And that goes against everything I’ve been doing in Erlang for ten years.

Do people really not use the functional patterns in Elixir? Heck, I’ve found it beyond hard pressed to get anything self-written to perform faster than Enum.map (the example we were using) and even than it was less than 1% speed difference with a substantially larger amount of code.

Is this really true for people? I use Enum extensively (occasionally even drop down to the base erlang function just because familiarity and the arguments are in the right order (*cough*callback*funs*first*cough*), but even that is waning as Enum can do about everything). So someone saying that Enum is not generic enough for iteration is mind boggling to me…

If Enum (and other similar functional things like for-comprehensions) are not really used (outside my own work apparently?), do people really ‘roll their own function loops’ that often? Sure they are easy but they harder to follow than a simple Enum |> chain, harder to read, breaks the flow of data up significantly more, and slower in every benchmark we passed between compared to Enum on Erlang OTP 19 (the hand-rolled ones were from between 5% to 9% slower than the single-line Enum’s on my machine with repeated tests, Ubuntu 16.04 latest everything).

So yeah, do people really not use Enum/for-comprehensions and instead do it manually? I’ve been doing erlang for almost ten years and other functional languages for a decade before that and I would never consider doing such a thing, even if for a 10% speed bonus unless absolutely necessary, it is a code-reading-nightmare to me…

I’ve never seen that either and to me it goes against Elixir having a strong standard library—reminds me of Node where people roll their own utilities because the STL sucks. I’ve never seen people use stuff manually and all learning materials I’ve seen advocate Enum.

I have literally never heard this sentiment, and I spend an unreasonable amount of time in IRC, Slack, and the various more asynchronous communication platforms (this forum, google groups).

Obviously cases exist where it isn’t the best option, but I have never heard someone recommend against it as a general rule.

As a side note, I’m not sure what you mean by slower. Enum.map for lists hands off to just a call to :lists.map which is built with the normal recursion w/ function application on each item. There’s no possible way a hand rolled function is slower. Hand rolled functions can be slightly faster IF AND ONLY IF you skip the anonymous function and hard code whatever function call you want to do at each item. This is rarely worth the extra code, but is good to know if it’s part of a tight loop.

Look at your experience: functional languages since two decades. Now consider developers coming from other languages, even hybrids like Scala or C# - the first thing everyone does is asking, where is for/foreach?

I personally have slowly moved into the functional mindset, but even a simple map-reduce in JavaScript for a MongoDB query was too much for many of my colleagues in projects. Universities or colleges are - at least in my experinece in Europe - not really into functional concepts and once people work they get on a track and never go away from it.

Universities or colleges are - at least in my experinece in Europe - not really into functional concepts and once people work they get on a track and never go away from it.

Same in the US—we only did for loops except in an elective where we explored different paradigms.

I think between the higher order functions and list comprehensions (which is just a very fancy for_each), doing a loop in Elixir should be rare.

Actually our benchmarking did find fascinating results here.

A hand-rolled loop is faster than Enum.map on Erlang OTP 18 by about 10%, but on Erlang OTP 19 Enum.map was faster by about 5-8% consistently on multiple runs. :slight_smile:

Only reason to avoid Enum I can think of, would be if you’re doing a lot of different kinds of modifications that could happen in the same loop. Instead of iterating multiple times if you were to use multiple Enum calls. e.g. If you wanted Enum.map + Enum.reduce + Enum.filter, while Enum does provide some combination operations it’s possible they have some use cases that have extra operations that don’t fit into the standard combination offerings Enum provides.

I think function composition and/or list comprehensions could solve these issues. The reason we’re confused is because the original people say they never use Enum.

Can you gist the benchmarking code? Enum.map isn’t a NIF, so all it is is a manually recursive function, so it doesn’t make any sense to compare Enum.map vs a recursive function, they’re both recursive functions.

To elaborate:

https://github.com/erlang/otp/blob/maint/lib/stdlib/src/lists.erl#L1240 is the :lists.map implementation, which Enum.map dispatches to when called with a list. Do note that it’s a body recursive function, so possibly if you were writing a tail recursive function you were observing that body recursion can be better at certain lengths.

This observation is in keeping with posts like http://ferd.ca/erlang-s-tail-recursion-is-not-a-silver-bullet.html

The ‘last’ version of it was this, using his hand-rolled version compared to enum, it requires Benchee.

list = Enum.to_list(1..100_000)
map_fun = &Kernel.*(2, &1)

defmodule EnumBench do
  def map(list, transformer) do
    list
    |> :lists.reverse
    |> map(transformer, [])
  end

  defp map([ ], _, acc), do: acc
  defp map([ head | tail ], transformer, acc),
    do: map(tail, transformer, [ transformer.(head) | acc ])
end

Benchee.run %{time: 10, warmup: 10, parallel: 1}, [
  {"Enum",
   fn -> Enum.map(list, map_fun) end},
 {"rolled",
   fn -> EnumBench.map(list, map_fun) end}
]

TCO was not even talked about, it was purely them stating that the Enum usage was less idiomatic and slower. We both tested and found the enum version slower on OTP18 and faster on OTP19, but both were less than 10% either way so it did not matter which was chosen, so it ended up coming down to ‘which was more commonly’ used, of which I asserted Enum and he asserted it was not, which was making me doubt everything I’d seen in the Elixir community but then again I’ve only really been on these forums before. ^.^

I think we can agree that Enum usage should be and is considered to be idiomatic.

The point I’m trying to make about the benchmarks question is that Enum.map isn’t special. Talking about “hand rolled” functions vs “Enum.map” doesn’t actually mean anything. If you write a body recursive function that take a list and a 1 arity function, you’ve got literally the same thing as Enum.map, they’re in every respect identical.

Put another way, you phrased it as though the benchmark was comparing “hand rolled” vs “Enum.map”. Really though you were benchmarking “tail recursive” (which your code is) vs “body recursive” functions generally in OTP 18 vs OTP 19. The BEAM doesn’t care whether the function is in the standard library or not.

Yes, but we were not caring about ‘how’ it was done (a body recursive version appeared too for example, that was just the latest version), it was just a thing of whether people use Enum or not so it was an example of Enum as I use it and a function that he himself had made, of styles (and readability in my opinion), not implementation details.

Let me amend @benwilson’s answer and say that building a custom recursive function and using Enum.reduceare pretty much identical. The way it is implemented, Enum.map is just a specialized version of reduce. (for purists: Enum. mapis not a ‘true’ map as it throws away the enumerable’s structure during iteration and always returns a list.)

Using the Enumerable protocol does add a layer of indirection, as Enumerable.reduceis implemented in a continuation-style way which also allows lazy enumeration using Stream.

Usually, this difference is far too small to make a difference. If you want to be sure, make your own benchmarks, but I would advise to use Enum whenever possible as it also greatly reduces code complexity and thus keeps your code easier to maintain.

The only times I would roll my own, are cases where you sometimes have to conditionally look ahead multiple elements, such as when writing a parser. In all other cases: Enum is more idiomatic and flexible.

Another great reason for using Enum is that it allows you to take any enumerable as input without rewriting your recursion.

I am very happy that my university teaches Haskell as a mandatory course, because there are so many problems that can be solved in such a clean way using functional programming.

That is what I was saying. He was saying that Enum is not generic enough yet could not give any examples or reasoning as to how, it just made no sense to me… >.>