Elixir vs Go Performance (Anton Putra)

Some ideas:

• setting +sbwt none to avoid busy wait consuming the allocated CPU slice (might be irrelevant in this benchmark if there is no wait under load) it was already set
• ensuring System.onilne_schedulers is equivalent to the available CPUs inside the container should be done automatically already: OTP 23 Highlights - Erlang/OTP
• ensuring TCP socket options (recbuf (rcvbuf), sndbuf, backlog, reuseport, etc.) are the same for Go and Elixir
• switching to :json
• (possibly) disabling Logger or increasing the log level (is Go logging?)
• separating the Web and DB benchmarks to see where the slowdown is coming from
• (possibly) spawning processes handling the requests / connections with larger heaps to avoid reallocs
• (possibly) switching from Bandit to an alternative web server, I remember playing with an Elli-like HTTP/1.1 server and it was outperforming both Bandit and Cowboy in benchmarks with wrk
• (possibly) switching from Postgrex to an alternative PostgreSQL client and/or connection pool
• (possibly) running multiple Postgrex pools

However, all of these steps are not common among Elixir developers, so I think the benchmarked app is about as representative (of what we typically write) as it gets. And the benchmark itself highlights some areas for possible improvement

The issue is not Postgrex, but rather the fact it is comparing Ecto (which would be akin to a ORM):

https://github.com/antonputra/tutorials/blob/59d29deeb57b1a81fe9c2c5e4dcb44e96c6bedb8/lessons/230/elixir-app/lib/app/router.ex#L62-L73

With something that directly sends INSERT commands over the connection:

https://github.com/antonputra/tutorials/blob/main/lessons/230/go-app/device.go#L30-L33

And if you are going to set up 500 database connections, I would at least split over a few connection pools (by setting pool_count). Otherwise you are likely making the pool the bottleneck.

Overall, I’d expect the Go to have better throughput, but there is definitely a bit of apples to oranges going on. I honestly don’t understand why benchmark authors do not ask for some community vetting before publishing.

The first test where it was mentioned that go uses a performant third-party JSON library vs using Jason in elixir is also geared towards putting go in a better light.

If raw performance is the main scope, then he could have used a library like jiffy or to make the comparison fair (since using NIFs can be called cheating), only limit to standard library.

What responses to that video would you have liked to see?

To re-iterate few points , I am very new to elixir
i was expecting elixir to be slower, but to do better on availability

So i was expecting replies on how to improve the benchmark where elixir should naturally do better than Go , which is availability

In other words, what is the trade off, what are we sacrificing performance for?

Are you not satisfied with the answer that more DB connections should be given in the pool?

If this current implementation handles 10k req/s comapred to 60k/s on golang side, how can it by design provide availability? I am missing something on how that measurement is done?

Set the timeout on requests to longer times and you will receive the responses, as the nature of how schedulers work in erlang is that the more processes you have to handle (in this case 1 connection = 1 process most probably), the slower all the system becomes.

IMO Elixir is much better than Go for a backend, and the thing all these benchmarks miss is the reality of how apps are actually used. The real environment a backend operates in can be thought of as rough terrain, while a benchmark environment is a paved road. These benchmarks take Elixir (a Land Cruiser SUV) and Go (a Corolla), measure how fast the car can go one mile burning x amount of fuel, and conclude the Corolla is a better car because the numbers are better. Meanwhile people who actually have to get over rough terrain drive the Land Cruiser.

There’s a highly upvoted post on the Go subreddit about a nil deference crashing the entire app and causing $100k in lost revenue. https://www.reddit.com/r/golang/comments/18sncxt/go_nil_panic_and_the_billion_dollar_mistake/

It seems extremely unlikely for a similar programming mistake to take down an Elixir app.

A couple more differences after looking into the serialization aspect:

1. They are using Jason.encode! instead of the more efficient Jason.encode_to_iodata!. Doing this change makes it 30% faster on my machine and drastically reduces memory allocation (this is what Phoenix’ json would have done by default)

2. They are not using the derived App.Device when encoding, which means the shape is not preallocated

3. The Go version encodes the current time once. The Elixir version encodes it 4x (twice to send it to the database, twice for JSON). Those things tend to cause a large difference in benchmarks, I replaced it here and it brought a 10%-20% improvement

Honestly, I don’t understand why it suddenly falls over in the first test. However, without a way to reproduce it, it is impossible for me to answer it. I cannot reproduce it on my machine. I am confidently pushing over 80k req/s (while Go sits at 110k).

Here is a pull request that makes the Elixir code closer to the code in Go and uses better defaults (for example, what Phoenix would have used): Make benchmarks with Elixir fairer by josevalim · Pull Request #370 · antonputra/tutorials · GitHub