What is the best BEAM server?

I was surprised to learn that WhatsApp dropped beefy machines and now run on servers capped at 32GB of RAM:

They also no longer run FreeBSD:

It wasn’t for performance tho, so I’d be interested in hearing more about optimal server hardware and OS combos as well (tho from what WhatsApp have done, it’s probably safe to say you don’t ‘need’ beefy machines to run very high traffic services :D)

But the main reason they did so was to fit in the Facebook DevOps model, not because was best. In fact to make it work they end-up to make a lot of contributions to the Erlang core and they still have some hacks that weren’t accepted into the core.

Yup.. but the takeaway is that Erlang can run huge-scale apps on less powerful hardware too (which could be more cost effective than fewer beefier machines).

I’d still be interested in any tests people have done to come up with optimal hardware specs/set-ups.

How can it be that?

Core cost scales more or less linearly, same for mem. Both onprem and in cloud. However bandwith cost will always increase the more nodes you have.

In that regard fewer nodes equalling same core and memory should always be cheaper if your work is parallellizable across all the cores, which on the beam is usually true.

Memory cost only scales linearly to a point in a single node - for instance 2x64GB DIMMs are somewhat cheaper than a single 128GB DIMM, and 4x64GB are much cheaper than a single 256GB. The bigger ones still sell, because there’s only so many slots in a single server motherboard.

If you run your own datacenter or you are co-locating then bandwidth per box shouldn’t be an issue - or am I missing something? (I’m speaking purely of dedicated servers, not cloud based hosting.)

(Dedicated) Server cost goes up noticeably after the higher-mid range, hence I’d be interested in knowing what would be better for an Erlang/Elixir app - 2 Hexa-Core Coffee Lake servers with 64GB ram each vs a single 18 core Cascade Lake with 128GB.

There’s also redundancy with additional servers (depending on how they were being used and whether it would be one big vs multiple smaller).

I was talking virtualized on that part, but also relevant for dedicated if you dont want all eggs in same basket though. More servers means more individual crosschatter across the mesh etc.

I was just making a hyptothetical point, the way you put your lego pieces of hosting together will rarely impact cost in meaningful way

I’m not so sure Oliver

Even just with dedicated servers the difference between 1 large vs 2 smaller, or 2 large vs 3 or 4 smaller could be big enough to consider smaller (and again if you have a whole datacenter full of them).

But what’s more performant could influence the decision if the difference is more than just slight… so if anyone does do such tests, I for one would be interested in hearing your findings

I wonder if @OvermindDL1 did any further experiments on that set-up^^?

Thats comparing lego with a sack of plastic raw material though

Since a many-core server would need a huge memory and I/O bandwidth between all cores and/or chiplets I’d wager that the best many-core-friendly setup is either an EPYC server with 32 / 64 / 128 cores or one of the ARM platforms with 80 cores (sorry, forgot the name ). They have insanely quick inter-bus communication so even with 100+ Erlang schedulers (one per CPU thread) the parallel overhead would be minimal to non-existent.

Or you can get the Ryzen 5800X3D since it has humongous L3 cache which probably helps a lot to not have CPU cache misses all the time.

Answering OP directly: yep, that would be an awesome server for Erlang/Elixir. Although I think the CPU power will be kind of wasted there, you could probably get away with less GHz and maybe even less cores. The rest looks ideal, especially inter-bus comms.