ancatrusca

ancatrusca

BEAM There, Done That with Chris Beck - Why ETS and a single Rust NIF beat Go, Rust, and Node for a BlueSky-scale workload

New BEAM There, Done That episode worth a read for anyone who’s had to defend an Elixir choice against teams pushing for Go, Rust, or Node on benchmark grounds alone.

Chris Beck (Bitcrowd, Berlin) walks through building an open source alternative to BlueSky’s data plane: the part of the AT Protocol stack that turns the firehose into an actual timeline a mobile app can render. Target: millions of users, ~10,000 messages/second, with real spikes already observed (six million active users during the November 2024 US election).

The team genuinely expected to land on Go, Rust, or Node, all four being languages they use daily. Each got ruled out for structural reasons rather than raw speed: Go’s GC pauses would have been a permanent tuning problem, Rust’s safety guarantees came at the cost of iteration speed they needed for a side project doing constant experimentation, and Node’s single-threaded scheduler is exactly why BlueSky’s own production team already moved away from their reference implementation.

What made Elixir work despite losing almost every microbenchmark:
ETS handled storage and follow-graph lookups at C speed without needing a NIF, covering most of what would otherwise have been Elixir’s weak spot. For the one genuinely hard case, bitmap computation for the “social proof” / mutual-follows feature, they reached for a Rust NIF via Rustler. They’re candid about the cost: a NIF runs in the BEAM’s own memory and a crash there can take the whole VM down, unlike a regular Elixir process failure.

The fan-out architecture, pushing messages directly into user mailboxes on arrival, gave them natural backpressure and burst absorption with no external queue needed, something they note would have added network and serialization overhead in any of the other three stacks.

Chris’s one-sentence takeaway for anyone choosing a stack: ignore what seems obvious on paper, start from a shopping list of actual requirements, and look at what genuinely cannot be solved any other way before deciding.

The project is open source, documented at bitcrowd.dev, and the team is actively looking for collaborators.

Most Liked

aseigo

aseigo

They specifically note the decision didn’t come down to raw performance: “What made Elixir work despite losing almost every microbenchmark”, “Each got ruled out for structural reasons rather than raw speed”.

For pretty much all the language options they listed, one could make it work with enough infrastructure and effort. And they said exactly that in the podcast, more than once! They even specifically noted that go was known to be a workable option since Bluesky uses it for their implementation.

Their decision came down to Elixir having solved large parts of their problem already and being well suited to their use case out of the box.

Rather than seeing this as “the other options can’t/don’t/won’t work well enough” (which is a pretty pessimistic / negative read of the presentation imho), it’s far more interesting to see that Elixir does work for these use cases at this scale.

A real-world application where people made thoughtful decisions and it worked out well for them with Elixir is great to see. Huzzah.

halturin

halturin

>Go’s GC pauses would have been a permanent tuning problem

Elixir Beat *

Sorry, but this is complete nonsense.

The claim that Go GC pauses would be some kind of permanent tuning problem is not supported by actual measurements. Even a simple ping benchmark from Ergo Framework GitHub - ergo-services/benchmarks: Benchmarks of the Ergo Framework 3.0 (and above) · GitHub demonstrates roughly 4.5 million messages per second over the network, which already contradicts the narrative that GC overhead makes this kind of throughput impractical.

And if anyone wants to see how such systems behave in a real distributed environment rather than in synthetic theorycrafting, take a look at the observability example:
https://github.com/ergo-services/examples/tree/master/observability

You can run a cluster in Docker, watch nodes, processes, network activity, supervision trees, and metrics in real time, and discover a completely different level of observability than what is usually discussed in these comparisons.

Benchmarks are one thing. Running, inspecting, and operating a live distributed cluster is another.

dimitarvp

dimitarvp

Looks like @aseigo beat me to it but I believe the real issue with Go (and practically every other non-BEAM language really) is that you’d have to reinvent half (or all) of OTP to get certain desirable app features and properties.

As a guy who works with Elixir, Golang and Rust, I like the latter two a lot (come on, try to achieve exhaustive pattern matching in Elixir! I know it’s coming but it might be coming for another year or two still). But try to coordinate thousands of micro-agents transparently and it’s becoming hugely laborious.

One negative of the Golang system is that it actually has excellent libraries, including the Ergo framework (funny I am saying this to you, it seems like you are the founder of the company that created it and maintains it? :thinking:). But in every Golang interview I was, and in every Golang team, devs always frown upon libraries and frameworks and always want the code to be “pure”. Which is really sad because Golang has a good amount of f.ex. channel footguns and goroutine leaks potential, and I’ve used multiple world-class libraries and frameworks (Ergo included). No idea why the wider Golang community is so conservative.

And, as much as I’d always advocate for Rust, its excellent compiler still cannot prevent you to deadlock an entire big circle of agents, each waiting on the next, until the last one turns out to wait on the first one. Trivial to do.

(Though in fairness, you can deadlock GenServers as well.)

Where Next?

Popular in Podcasts Top

brainlid
In episode 57 of Thinking Elixir, we sat down with co-host Cade Ward to hear how he and his team tackled a problem of hosting live web ch...
New
brainlid
In episode 54 of Thinking Elixir, Lucas San Román explains his library Sourceror and how it was created to solve some AST parsing limitat...
New
brainlid
In episode 71 of Thinking Elixir, we catch up with German Velasco then talk about the 7-GUI Tasks online challenge and how he wanted to s...
New
brainlid
Episode 161 of Thinking Elixir. Language Servers underpin the language specific support we rely on in modern code editors. Lately, there ...
New
brainlid
Episode 222 of Thinking Elixir. News includes the release of OTP 27.1 with significant improvements to the Zip module, the upcoming depre...
New
brainlid
In episode 72 of Thinking Elixir, we talk with Wiebe-Marten Wijnja about his project TypeCheck that provides a new kind of runtime type c...
New
brainlid
https://thinkingelixir.com/podcast-episodes/051-live-auctions-with-alex-loukissas Posted via Devtalk (see this thread for details).
New
brainlid
Episode 181 of Thinking Elixir. In this week’s episode, we ignite the exciting world of Elixir with Chris McCord’s announcement of FLAME,...
New
brainlid
In episode 75 of Thinking Elixir, we talk with Dave Lucia about Simplebet’s use of RabbitMQ and Commanded for solving unique real-time pr...
New
ancatrusca
Task.async(fn → Rust.performant() end) Elixir :handshake: Rust What actually happens when BEAM resilience meets Rust performance? Flor...
New

Other popular topics Top

axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 48475 226
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29703 241
New
greenz1
I have a phoenix application from which a user can download multiple(5-6) files of size 1MB. I couldn’t find anything related to sending ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 records...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
dogweather
I wrote this comment on r/haskell, and it’s not popular there. :wink: But I think I’m on to something… Haskell reminds me of Java, and e...
New

We're in Beta

About us Mission Statement