Enum fusion?

Are you encountering a speed bottleneck somewhere?

What a cool idea! I would love to see how far this idea could go

No. This is an ‘how far could we take it and would it be worth it’ rather than a ‘my code is too slow’ kind of situation.

My suggestion is write it and measure. My instinct is that when you find a domain where there’s a performance improvement over Enum, (let’s say, 10% better than enum) you are in a regime where the Stream penalty is worth it, and you’re say… only 2% better than Stream.

Ultimately, you will be hamstrung by speed of the erlang datatypes. If you need better, you should write a nif or something. But really I don’t understand the obsession with performance. For most domains where you’re using Elixir, other things, like network or database, dominate your performance concerns.

I think you are right - most Elixir apps will have bottlenecks elsewhere (as OP pointed out), but it is very healthy, I think, to try and squeeze more out of a platform where possible. Each little tweak contributes to keeping hosting costs down, reducing energy consumption and improving user experience. It’s also fun!

You mean Stream?

No. Stream is lazy.

What I mean is that something like

mycollection
|> Enum.map(&foo/1)
|> Enum.map(&bar/1)
|> Enum.map(&baz/1)
|> Enum.reduce(starting_accumulator, &qux/2)

can be turned into

for x <- mycollection, reduce: starting_accumulator do
  acc -> 
    x2 = x |> foo() |> bar() |> baz()
    qux(x2, acc)
end

(and this can be expanded further to e.g. also work with filter, flat_map, into, etc.)

Which is lazy evaluation with forced computation at the end of pipeline:

mycollection
|> Stream.map(&foo/1)
|> Stream.map(&bar/1)
|> Stream.map(&baz/1)
|> Enum.reduce(starting_accumulator, &qux/2)

Does (almost) exactly what you want.

That’s not quite the same.

The benefit of proper fusion is to get the memory characteristics of the Stream (which avoids building and intermediate lists) while retaining the speed of a regular Enum.

In combination a fused Enum is therefore faster in theory as it avoids allocations and takes stress from the GC.

Performance is not a topic that be completely ignored. Elixir is not the language to choose for a number crunching application, but sometimes you need to do some number crunching as part of a larger application and the benefits of not having to go down the NIF or Port route are enormous. There’s a reason the VM team are implementing a JIT for the next version.

In the Elixir project that I have just finished (telecoms hardware), the vast majority of the code is not performance sensitive at all (and sometimes latency sensitive like a Web app is). There is one (key) bit of functionality that allocates time slots on the link. It’s a difficult algorithmic problem (basically the “knapsack problem” with some extra constraints). The implementation in Elixir is a little slow (3 to 30 seconds per link). We did a bit of optimisation and we were able to simplify the problem a bit to ensure that that it ran within the available time. The simplification might not have been possible and we would have needed to optimise further and would have been happy to be non-idiomatic. Something like this library might have helped. I would be curious to see how much speed be fit this approach would actually provide.