Benchmark report the-benchmarker and phoenix

I’ll be the sour grape, hold tight.

Most benchmarks, especially those of web frameworks, are deeply flawed. It’s a game of numbers, and when ranks and numbers are involved, people are bound to try and cheat their way to the top ranks. The most popular benchmark – TechEmpower – is widely known to not enforce good benchmark discipline in the accepted code so instead of having proper HTTP servers you have plain text processors that accept hard-coded HTTP headers (one of the egregious examples).

This has skewed results significantly to the point that many developers only take a cursory glance at such benchmarks and leave almost immediately.

Is this benchmark framework better? Maybe it is, but who is going to use their work time (and their employer does not pay them to go research if a benchmark framework is good) or their free time in order to establish this? Not saying that nobody will do it – pretty sure somebody will. But it might be too much effort just to uncover yet another benchmark framework that lets people cheat their way to the top ranks.

I knew the case of techEmpower and I think the same as you, this type of benchmarks, in which there is a rank and multiple frameworks, are not usually reliable examples.
Thanks you for your point of view.
Best regards.

That’s what I just did

We definitely need better benchmarks though…

A proper benchmarking system (like the one I have in mind) could be a huge win for Elixir/Erlang

IMO this benchmark is entirely unrealistic; the top performers are doing exactly what’s required, but leaving off 95% of what most applications would actually need. For instance, take a look at the “top performer” ActiveJ’s approach:

https://github.com/the-benchmarker/web-frameworks/blob/332e59bc6e416f5b7002b5dea5511cad05910ba8/java/activej/src/main/java/benchmark/io/activej/http/Main.java#L28-L31

Unless I’m missing some magic in RoutingServlet, this responds to requests with the ABSOLUTE MINIMUM required to satisfy the requirements; HTTP/1.1 200 OK with no headers at all, then either EOF or a few bytes of content (for the /user/:id route).

The Phoenix benchmark, OTOH, sends more headers and does more work.