Qqwy

Qqwy

TypeCheck Core Team

Hi all!

As you may know, it is very common in idiomatic Elixir code to work with transformations on datastructures, especially those implementing the Enumerable protocol. Usually we use the functions from Enum (and sometimes from Stream) to manipulate these, often in a pipeline of multiple steps.

Since the Enumerable protocol is an implementation of the Foldable concept from category theory, whose main fundamental operation is reduce(in other languages also known as ‘fold’), which always outputs a list, we end up using lists virtually everywhere.

This means that in an Enum pipeline, a lot of intermediate lists are being generated. I seem to remember José explaining on the mailinglist back in the day(note: I was unable to find it; if you know where this was mentioned, let me know and I’ll link to it!) that Enum’s functions were intentionally not implemented as macros to make it easier to follow stack traces when something broke.

I think this is definitely the right choice, especially since a lot of Elixir code is written with “IO-bound” operations in mind, in which sheer computing speed is less important.

However, it did start making me wonder: What about creating a FastEnum drop-in replacement, where map, reduce, etc. would be implemented as macros that would fuse consecutive operations together to improve performance?
In many cases, a pipeline of Enum-functions could be transformed into a single for-comprehension. Besides the added benefit of fusing consecutive calls, for is also extremely well optimized by the BEAM.

Now, why did I start this topic? I essentially have two questions/topics for discussion:

  • Do you think a library like this would be worthwhile?
  • Do you happen to know whether someone already performed any exploratory work in this direction? (The ideas presented here are, after all, far from novel.)

Showing Posts 1 to 10

ityonemo

ityonemo

Are you encountering a speed bottleneck somewhere?

lpil

lpil

Creator of Gleam

What a cool idea! I would love to see how far this idea could go

Qqwy

Qqwy OP

TypeCheck Core Team

No. This is an ‘how far could we take it and would it be worth it’ rather than a ‘my code is too slow’ kind of situation.

ityonemo

ityonemo

My suggestion is write it and measure. My instinct is that when you find a domain where there’s a performance improvement over Enum, (let’s say, 10% better than enum) you are in a regime where the Stream penalty is worth it, and you’re say… only 2% better than Stream.

Ultimately, you will be hamstrung by speed of the erlang datatypes. If you need better, you should write a nif or something. But really I don’t understand the obsession with performance. For most domains where you’re using Elixir, other things, like network or database, dominate your performance concerns.

mindok

mindok

I think you are right - most Elixir apps will have bottlenecks elsewhere (as OP pointed out), but it is very healthy, I think, to try and squeeze more out of a platform where possible. Each little tweak contributes to keeping hosting costs down, reducing energy consumption and improving user experience. It’s also fun!

hauleth

hauleth

You mean Stream?

Qqwy

Qqwy OP

TypeCheck Core Team

No. Stream is lazy.

What I mean is that something like

mycollection
|> Enum.map(&foo/1)
|> Enum.map(&bar/1)
|> Enum.map(&baz/1)
|> Enum.reduce(starting_accumulator, &qux/2)

can be turned into

for x <- mycollection, reduce: starting_accumulator do
  acc -> 
    x2 = x |> foo() |> bar() |> baz()
    qux(x2, acc)
end

(and this can be expanded further to e.g. also work with filter, flat_map, into, etc.)

hauleth

hauleth

Which is lazy evaluation with forced computation at the end of pipeline:

mycollection
|> Stream.map(&foo/1)
|> Stream.map(&bar/1)
|> Stream.map(&baz/1)
|> Enum.reduce(starting_accumulator, &qux/2)

Does (almost) exactly what you want.

NobbZ

NobbZ

That’s not quite the same.

The benefit of proper fusion is to get the memory characteristics of the Stream (which avoids building and intermediate lists) while retaining the speed of a regular Enum.

In combination a fused Enum is therefore faster in theory as it avoids allocations and takes stress from the GC.

Will-W

Will-W

Performance is not a topic that be completely ignored. Elixir is not the language to choose for a number crunching application, but sometimes you need to do some number crunching as part of a larger application and the benefits of not having to go down the NIF or Port route are enormous. There’s a reason the VM team are implementing a JIT for the next version.

In the Elixir project that I have just finished (telecoms hardware), the vast majority of the code is not performance sensitive at all (and sometimes latency sensitive like a Web app is). There is one (key) bit of functionality that allocates time slots on the link. It’s a difficult algorithmic problem (basically the “knapsack problem” with some extra constraints). The implementation in Elixir is a little slow (3 to 30 seconds per link). We did a bit of optimisation and we were able to simplify the problem a bit to ensure that that it ran within the available time. The simplification might not have been possible and we would have needed to optimise further and would have been happy to be non-idiomatic. Something like this library might have helped. I would be curious to see how much speed be fit this approach would actually provide.

Where Next? Top

Trending in Discussions Top

AstonJ
The obligatory hello world thread! Who are you and where are you from? :stuck_out_tongue:
4663 61218 595
New
cblavier
Hey there, It’s been more than a year since we started using LiveView as our main UI library and building a whole library of UI componen...
515 12931 101
New
raa-ziq
Hi everyone :waving_hand: I wanted to share something I’ve been building — NowBlind, a random chat app designed to help people discover ...
New
sntran
In my third year of college, around 2010, I found Erlang. A few years later, I found Elixir. I liked the ideas at once: many small proces...
New
gtcode
I’ve been tinkering with Lynx (by José) via hold_ledger. I used AI to write the code, a separate specification, the tests, and the Lean p...
New
AstonJ
A remark @Garrison made… ..has already inspired a thread for the technical side of reclaiming our internet, but there is of course anot...
New
ManuelGarcia
Hi everyone! I’m the developer of PaperForge, and I’ve been working on something alongside it that I’d like your opinion on. I’m conside...
New

Other Trending Topics Top

dominicletz
Hey there, I haven’t seen a list of useful Elixir specific skills for the popular AI tools yet so I’m starting one here with my first. ...
New
mudasobwa
I fully migrated to my own harness from Anthropic/Gemini and I think it’s time to share it. Welcome DSH, the DeepSeek Harness, fully writ...
New
OndrejValenta
(I just needed to vent somewhere and LinkedIn is full of hope, or hype, I’m not sure which exactly) AI dream has many faces, but general...
New
kip
Localize is the next generation localisation library for Elixir. Think of it as ex_cldr version 3.0. The first version will be released ...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
mcass19
ExRatatui lets you cook up rich terminal UIs in Elixir, powered by Rust’s ratatui via Rustler NIFs. Build interactive terminal applicatio...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews