ancatrusca

ancatrusca

BEAM There, Done That with Chris Beck - Why ETS and a single Rust NIF beat Go, Rust, and Node for a BlueSky-scale workload

New BEAM There, Done That episode worth a read for anyone who’s had to defend an Elixir choice against teams pushing for Go, Rust, or Node on benchmark grounds alone.

Chris Beck (Bitcrowd, Berlin) walks through building an open source alternative to BlueSky’s data plane: the part of the AT Protocol stack that turns the firehose into an actual timeline a mobile app can render. Target: millions of users, ~10,000 messages/second, with real spikes already observed (six million active users during the November 2024 US election).

The team genuinely expected to land on Go, Rust, or Node, all four being languages they use daily. Each got ruled out for structural reasons rather than raw speed: Go’s GC pauses would have been a permanent tuning problem, Rust’s safety guarantees came at the cost of iteration speed they needed for a side project doing constant experimentation, and Node’s single-threaded scheduler is exactly why BlueSky’s own production team already moved away from their reference implementation.

What made Elixir work despite losing almost every microbenchmark:
ETS handled storage and follow-graph lookups at C speed without needing a NIF, covering most of what would otherwise have been Elixir’s weak spot. For the one genuinely hard case, bitmap computation for the “social proof” / mutual-follows feature, they reached for a Rust NIF via Rustler. They’re candid about the cost: a NIF runs in the BEAM’s own memory and a crash there can take the whole VM down, unlike a regular Elixir process failure.

The fan-out architecture, pushing messages directly into user mailboxes on arrival, gave them natural backpressure and burst absorption with no external queue needed, something they note would have added network and serialization overhead in any of the other three stacks.

Chris’s one-sentence takeaway for anyone choosing a stack: ignore what seems obvious on paper, start from a shopping list of actual requirements, and look at what genuinely cannot be solved any other way before deciding.

The project is open source, documented at bitcrowd.dev, and the team is actively looking for collaborators.

Most Liked

aseigo

aseigo

They specifically note the decision didn’t come down to raw performance: “What made Elixir work despite losing almost every microbenchmark”, “Each got ruled out for structural reasons rather than raw speed”.

For pretty much all the language options they listed, one could make it work with enough infrastructure and effort. And they said exactly that in the podcast, more than once! They even specifically noted that go was known to be a workable option since Bluesky uses it for their implementation.

Their decision came down to Elixir having solved large parts of their problem already and being well suited to their use case out of the box.

Rather than seeing this as “the other options can’t/don’t/won’t work well enough” (which is a pretty pessimistic / negative read of the presentation imho), it’s far more interesting to see that Elixir does work for these use cases at this scale.

A real-world application where people made thoughtful decisions and it worked out well for them with Elixir is great to see. Huzzah.

halturin

halturin

>Go’s GC pauses would have been a permanent tuning problem

Elixir Beat *

Sorry, but this is complete nonsense.

The claim that Go GC pauses would be some kind of permanent tuning problem is not supported by actual measurements. Even a simple ping benchmark from Ergo Framework GitHub - ergo-services/benchmarks: Benchmarks of the Ergo Framework 3.0 (and above) · GitHub demonstrates roughly 4.5 million messages per second over the network, which already contradicts the narrative that GC overhead makes this kind of throughput impractical.

And if anyone wants to see how such systems behave in a real distributed environment rather than in synthetic theorycrafting, take a look at the observability example:
https://github.com/ergo-services/examples/tree/master/observability

You can run a cluster in Docker, watch nodes, processes, network activity, supervision trees, and metrics in real time, and discover a completely different level of observability than what is usually discussed in these comparisons.

Benchmarks are one thing. Running, inspecting, and operating a live distributed cluster is another.

dimitarvp

dimitarvp

Looks like @aseigo beat me to it but I believe the real issue with Go (and practically every other non-BEAM language really) is that you’d have to reinvent half (or all) of OTP to get certain desirable app features and properties.

As a guy who works with Elixir, Golang and Rust, I like the latter two a lot (come on, try to achieve exhaustive pattern matching in Elixir! I know it’s coming but it might be coming for another year or two still). But try to coordinate thousands of micro-agents transparently and it’s becoming hugely laborious.

One negative of the Golang system is that it actually has excellent libraries, including the Ergo framework (funny I am saying this to you, it seems like you are the founder of the company that created it and maintains it? :thinking:). But in every Golang interview I was, and in every Golang team, devs always frown upon libraries and frameworks and always want the code to be “pure”. Which is really sad because Golang has a good amount of f.ex. channel footguns and goroutine leaks potential, and I’ve used multiple world-class libraries and frameworks (Ergo included). No idea why the wider Golang community is so conservative.

And, as much as I’d always advocate for Rust, its excellent compiler still cannot prevent you to deadlock an entire big circle of agents, each waiting on the next, until the last one turns out to wait on the first one. Trivial to do.

(Though in fairness, you can deadlock GenServers as well.)

Where Next?

Popular in Podcasts Top

brainlid
In episode 68 of Thinking Elixir, we talk with Niklas Begley about his Doctave service and the problems it’s solving. From there we cove...
New
brainlid
News covers the new Phoenix.Template project, upcoming Nx work for Elixir using multiple GPUs, benefits of Elixir as seen by new people l...
New
wolf4earth
The Elixir Mix panel takes the helm to talk about helping onboard and transition new developers onto an Elixir team. They discuss helping...
New
brainlid
Episode 161 of Thinking Elixir. Language Servers underpin the language specific support we rely on in modern code editors. Lately, there ...
New
brainlid
In episode 62 of Thinking Elixir, we talk with Chris McCord about his recent announcement that he’s moved to work at Fly.io! We cover wha...
New
wolf4earth
Brian Underwood joins the mix to discuss his recent project where he created a game that would push more and more load onto a GenServer t...
New
brainlid
Day 1 and 2 of Livebook’s first launch week announcements, a new JsonSchema compiler named Exonerate, Ecto 3.10.0 brings changeset improv...
New
brainlid
Episode 111 of Thinking Elixir. Ever wanted to deploy a PR for show-and-tell, QA testing, or to get feedback on it? Jason Axelson joins u...
New
brainlid
Episode 129 of Thinking Elixir. Tracking, analyzing and visualizing time series data can add a lot of business value to a project! We met...
New
thiagomajesk
I saw this podcast episode on my Twitter timeline today and wanted to share it with you. I found a lot of valuable lessons packed in a 27...
New

Other popular topics Top

axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 48342 226
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29603 241
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
jason.o
In the code below, if the create action is not set to accept “extra_key” as an input, it errors out with a message shown above. Is there ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

Latest on Elixir Forum

We're in Beta

About us Mission Statement