I’d say this benchmark is borderline useless. There are three test cases which are not doing anything at all except a minimal amount of routing.

They test how fast the frameworks can read/write from/to a socket. It is also not specifying what headers need to be returned back which makes the comparison even more unfair. For example are date headers included? They can be quite expensive to calculate. The number of headers returned also change the size of the response which in a test like this may have quite an impact on the result as some of the frameworks might have to return double the amount of data depending on what headers they return.

The client is dubious as well and extremely simplistic. Instead something like wrk2 should be used which calculates percentiles and takes coordinate omission in account.

It also doesn’t specify hardware specifications for client server, the client is likely run on the same machine as the server, concurrency is low and the number of requests way too low. The test should run for a much longer period of time and run at various concurrency level.

The techempower web benchmarks are perhaps better but even they are artificial and test doesn’t run for long enough so that things like memory pressure and garbage collection can be taken into account. Lots of examples on these test has also been optimized for what they are being tested for which is usually a bad idea if you want a result that can be compared in the general case.

I love this community. Not only did y’all improve the reporting of benchmarks for Phoenix and Plug, but y’all also improved and added the benchmarks for other frameworks not even elixir related.

This is the same problem as with all benchmarking, what are you measuring and what is really, really, important for you. The fastest webserver would be one that whatever you send just returns some static lines, it will be really fast and beat anything, but in most cases totally useless. So the question is in what are you interested? Is it raw throughput? Or maybe massive concurrency? Or maybe low latency? Or maybe very versatile system? Or maybe high-reliability? Or maybe …?

Unfortunately, building a system usually entails making trade-offs. TANSTAAFL.

Another important issue is how you actually measure these things. Is your way of measuring interesting and relevant for what you want/need? Do you know what you need?

I will stop here but benchmarking is a fascinating field.