Benchmark report the-benchmarker and phoenix

[JuanjoA]

Hi, what do you think of phoenix’s position in these tests? (link)
I see many frameworks above in the list and I understant that it is not only the performance that is important but just to know what You think, which I’m sure you are objetive as I’m appreciating in other forum threads.

Thanks

[dimitarvp]

I’ll be the sour grape, hold tight.

Most benchmarks, especially those of web frameworks, are deeply flawed. It’s a game of numbers, and when ranks and numbers are involved, people are bound to try and cheat their way to the top ranks. The most popular benchmark – TechEmpower – is widely known to not enforce good benchmark discipline in the accepted code so instead of having proper HTTP servers you have plain text processors that accept hard-coded HTTP headers (one of the egregious examples).

This has skewed results significantly to the point that many developers only take a cursory glance at such benchmarks and leave almost immediately.

Is this benchmark framework better? Maybe it is, but who is going to use their work time (and their employer does not pay them to go research if a benchmark framework is good) or their free time in order to establish this? Not saying that nobody will do it – pretty sure somebody will. But it might be too much effort just to uncover yet another benchmark framework that lets people cheat their way to the top ranks.

[JuanjoA]

I knew the case of techEmpower and I think the same as you, this type of benchmarks, in which there is a rank and multiple frameworks, are not usually reliable examples.
Thanks you for your point of view.
Best regards.

[AstonJ]

@dimitarvp

This has skewed results significantly to the point that many developers only take a cursory glance at such benchmarks and leave almost immediately.

That’s what I just did

We definitely need better benchmarks though…

Benchmarks - how important are they to you? To Elixir?

Inspired by @atraac’s post here - how important are benchmarks to you and how important do you think they are to languages like Elixir? (Languages where performance and scalability are some of their biggest selling points.) As has come up in conversation in a few threads over the years, there seems to be a general frustration with benchmarks like the Techempower ones because they don’t really reflect real life usage. That’s particularly bad for Elixir and Erlang because real life situations is …

A proper benchmarking system (like the one I have in mind) could be a huge win for Elixir/Erlang

[al2o3cr]

IMO this benchmark is entirely unrealistic; the top performers are doing exactly what’s required, but leaving off 95% of what most applications would actually need. For instance, take a look at the “top performer” ActiveJ’s approach:

the-benchmarker/web-frameworks/blob/332e59bc6e416f5b7002b5dea5511cad05910ba8/java/activej/src/main/java/benchmark/io/activej/http/Main.java#L28-L31

Unless I’m missing some magic in RoutingServlet, this responds to requests with the ABSOLUTE MINIMUM required to satisfy the requirements; HTTP/1.1 200 OK with no headers at all, then either EOF or a few bytes of content (for the /user/:id route).

The Phoenix benchmark, OTOH, sends more headers and does more work.