As you may know, it is very common in idiomatic Elixir code to work with transformations on datastructures, especially those implementing the
Enumerable protocol. Usually we use the functions from
Enum (and sometimes from
Stream) to manipulate these, often in a pipeline of multiple steps.
Enumerable protocol is an implementation of the Foldable concept from category theory, whose main fundamental operation is
reduce(in other languages also known as ‘fold’), which always outputs a list, we end up using lists virtually everywhere.
This means that in an
Enum pipeline, a lot of intermediate lists are being generated. I seem to remember José explaining on the mailinglist back in the day(note: I was unable to find it; if you know where this was mentioned, let me know and I’ll link to it!) that
Enum's functions were intentionally not implemented as macros to make it easier to follow stack traces when something broke.
I think this is definitely the right choice, especially since a lot of Elixir code is written with “IO-bound” operations in mind, in which sheer computing speed is less important.
However, it did start making me wonder: What about creating a
FastEnum drop-in replacement, where
reduce, etc. would be implemented as macros that would fuse consecutive operations together to improve performance?
In many cases, a pipeline of
Enum-functions could be transformed into a single
for-comprehension. Besides the added benefit of fusing consecutive calls,
for is also extremely well optimized by the BEAM.
Now, why did I start this topic? I essentially have two questions/topics for discussion:
- Do you think a library like this would be worthwhile?
- Do you happen to know whether someone already performed any exploratory work in this direction? (The ideas presented here are, after all, far from novel.)