Just a note from the other numerous times TechMeme comes up, last year (or before) they added Phoenix in a preview, but it was a very poor implementation. They were testing JSON benchmarks through the :browser pipeline, complete with crsf token generation. They had a dev DB pool size of 10, where other frameworks were given of pool size of 100. And they also had heavy IO logging, where other frameworks did no logging. We sent a PR to address these issues, and I was hoping to see true results in later runs, but recent runs have show high error rates and we don’t know the details why nor have they done a great job with any of the recent code as shown in this thread. tldr; these results are not representative of the framework and the core-team’s time is better spent elsewhere. For those interested, please feel free to send PRs to improve their code, but it’s not something I lose sleep over

This is just not true. Our Chanel benchmarks do more than just accept a connection. We exercise the channel protocol which spawns and isolated, concurrent process monitored by the underlying websocket transport, which is subscribed to our pubsub layer. We also put the pubsub layer thru its paces by broadcasting messages to the 2M clients.

This is not a fair point. Saša has done nothing here to do mental dynamistics to make up for any apparent elixir or phoenix deficiencies. For anyone remotely adept at writing Elixir, the techmeme app is obviously poorly written, from configuration to middleware to lack of use of concurrency in the db tests, and more. My experience with them over the last two years has given me little confidence to see value in their results.

I’ll also note that this thread is turning unnecessarily negative, so please take care to avoid things going toxic

It looks OK (as expected) to me

From my point of view there are so many things that are off with the way they do these tests and they even say themselves that results are not comparable unless you are an “expert”.

From a quick glance:

1. They use wrk which has the coordinated omission problem. So basically their test result can only be interpreted for closed system (i.e not something where there is no back-pressure from clients like the internet)
2. They report the best case result for each framework. I think worst case is more interesting if I am running something in production
3. For latency they use average latency for ranking. It is well known that averages are bad to use in this case. They could have used proper percentiles or if they don’t do any coordinated omission adjustment (which they don’t) they should use the max latency.
4. Tests run for way too short. 15 seconds. Max concurrency 256 for anything but plaintext. They say themselves that they didn’t see any difference running test for 60 seconds or 15 seconds. Well I am not surprised:) 60 seconds is way too short as well! Once memory buffers run full, garbage collection kicks in, database connections get stuck you will see other problems. I’ve had servers that looked like they were running OK which ground to a halt and complete stop after 30 minutes. I understand it is hard to run the tests for all these frameworks for such a long time but still.

Are their tests useless? If you are contemplating setting up an http server in a closed system then perhaps no. They might be valid if you interpret the data correctly. If you have the http servers on an open system such as the internet? No then they are not valid.

And they say (Unfair comparisons / Tiamat):
“The tests set what can be considered high-water marks for performance.” That is, any framework is unlikely to perform better than these tests. So at least that is one thing you can takeaway from the results.

If you look through the data tables and latency graphs phoenix actually does pretty well. For latency it has low “max” and standard deviation which is not completely off the walls.

If you look how it performs under the various concurrency settings it performs as expected with little variation. Some of the faster frameworks fluctuated over 100% between the various concurrency settings. That is not what I call “stable”

Finally for these tests erlang VM has a bottleneck which hopefully will be addressed in OTP20 with multi poll-set. Not that it will be the best but I have a feeling it will do much better.

There is a danger with these tests that something like phoenix comes out in a bad light. There are python/php/javascript frameworks that come out on top in the way the present the data. I know from experience that in real life they stand no chance if you need performance out of the box.

I hope people don’t look at this and outright discards any framework because of it, but I think that is the case.