cmkarlsson

cmkarlsson

It looks OK (as expected) to me :slight_smile:

From my point of view there are so many things that are off with the way they do these tests and they even say themselves that results are not comparable unless you are an “expert”.

From a quick glance:

  1. They use wrk which has the coordinated omission problem. So basically their test result can only be interpreted for closed system (i.e not something where there is no back-pressure from clients like the internet)
  2. They report the best case result for each framework. I think worst case is more interesting if I am running something in production
  3. For latency they use average latency for ranking. It is well known that averages are bad to use in this case. They could have used proper percentiles or if they don’t do any coordinated omission adjustment (which they don’t) they should use the max latency.
  4. Tests run for way too short. 15 seconds. Max concurrency 256 for anything but plaintext. They say themselves that they didn’t see any difference running test for 60 seconds or 15 seconds. Well I am not surprised:) 60 seconds is way too short as well! Once memory buffers run full, garbage collection kicks in, database connections get stuck you will see other problems. I’ve had servers that looked like they were running OK which ground to a halt and complete stop after 30 minutes. I understand it is hard to run the tests for all these frameworks for such a long time but still.

Are their tests useless? If you are contemplating setting up an http server in a closed system then perhaps no. They might be valid if you interpret the data correctly. If you have the http servers on an open system such as the internet? No then they are not valid.

And they say (Unfair comparisons / Tiamat):
“The tests set what can be considered high-water marks for performance.” That is, any framework is unlikely to perform better than these tests. So at least that is one thing you can takeaway from the results.

If you look through the data tables and latency graphs phoenix actually does pretty well. For latency it has low “max” and standard deviation which is not completely off the walls.

If you look how it performs under the various concurrency settings it performs as expected with little variation. Some of the faster frameworks fluctuated over 100% between the various concurrency settings. That is not what I call “stable”

Finally for these tests erlang VM has a bottleneck which hopefully will be addressed in OTP20 with multi poll-set. Not that it will be the best but I have a feeling it will do much better.

There is a danger with these tests that something like phoenix comes out in a bad light. There are python/php/javascript frameworks that come out on top in the way the present the data. I know from experience that in real life they stand no chance if you need performance out of the box.

I hope people don’t look at this and outright discards any framework because of it, but I think that is the case.

7
Post #11
brightball

brightball

Part of the trap that we fall into is that people tend to assume “benchmarks + developer time = all factors”. We’re as guilty of it as anybody by showing the comparisons for the Ruby crowd but there’s always somebody faster. We need to do a better job clearly communicating the non-benchmark related perks of Elixir/Phoenix to clearly differentiate it for the casual observer.

2
Post #12
shortlyportly

shortlyportly

Hi All.

Comparing round 13 vs 14 it seems that Phoenix has gone backwards - does anyone have any idea why? I appreciate that these aren’t real world figures but I’d expect them to be consistent between the two rounds.

cheers

Dave

1
Post #14
kseg

kseg

You can look at this and say “we have room for improvement” or you can look at this and say “they don’t know how to benchmark.” One will lead to a better product, the other won’t.

When it comes to performance, Elixir/Phoenix/Ecto is a middle-of-the-pack player. For a dynamic language, it’s doing pretty well. But things could be improved. And, maybe some things can’t be improved because of the nature of Erlang.

1
Post #15
shortlyportly

shortlyportly

Thanks @kseg. I’m can’t disagree with you on your points. I was just curious as to why Phoenix seems to have slipped down the chart compared to previous versions (not against other frameworks). So, for example, The Data Updates test;

  • Round 12 - 1,100 (27.8%)
  • Round 13 - 1,915 (65.0%)
  • Round 14 - 750 (17.1%) - quite a significant drop.

cheers

Dave

1
Post #16
kseg

kseg

Not sure. Another reason I like these types of benchmark…even with all their “flaws”, if the “flaws” are consistent, they can catch regressions.

This commit changed the data update test:
https://github.com/TechEmpower/FrameworkBenchmarks/commit/96deafcf351ae0b7ae44ae51b292cfa829eab382#diff-92a2750f03cbfd63d2c3b27eeeef442a

You can see it no longer serializes through Poison’s @derive attribute which, according to the Poison docs, should be used for “maximum performance”.

There was also a more recent commit that changed many (faster) encode_to_dataio! to encode! which isn’t going to help anything.

2
Post #17
sasajuric

sasajuric

Author of Elixir In Action

While I agree that these perf tests can be useful to discover possible issues, I also agree with @cmkarlsson that these tests are a terrible way to compare frameworks, which is according to the site their main purpose (This is a performance comparison of many web application frameworks).

IMO with TE benches, you can’t really conclude whether one framework is faster than another for your particular problem. Moreover speed is not the only factor (nor the most important one) for choosing a web framework. Therefore, I think that these benches are very shallow and misleading, and that they fail in their main purpose.

On the other hand, having a common suite for benching frameworks would be great. If we could have an easy way of installing TE suite locally and running, we could have an automated way of benching our frameworks, finding out possible bottlenecks, and even detecting regression performance. Not sure to what extent can TE code currently be used for that, but I agree that this can be useful.

It’s worth noting that the goal of optimizing should be to get good and consistent performance numbers (latency and throughput), and not to move the framework up this shallow top list. Because, if the latter is the goal, we might end up doing all sort of trickery just to create the illusion that Phoenix is faster, introducing non-obvious trade-offs which will bite us in more complex real-life scenarios.

2
Post #18
kseg

kseg

It might be a bad way to compare frameworks, but it’s a pretty good way to compare the performance of HTTP servers and database drivers.

Personally, I think it’s safe to say that both net/http and Cowboy satisfy the HTTP specification, provide a usable interface to build frameworks on top of, are capable of scaling beyond a single core and have proven to be fairly robust. At that point, performance (latency, throughput, memory usage, …) is the only factor. At least, it’s the only factor from the point of view of people working on these components.

I agree that to a consumer, maybe speed isn’t the only factor. They are very different in another respect: one lets you run Go code, the other lets you run Erlang/Elixir code. Maybe that ought to weigh more on your decision than an X% difference in performance.

Or maybe not. I don’t know what your needs are, and I’m not going to assume that performance isn’t really important to you.

People will take whatever value from these benchmarks as they want. Yes, a lot of that will be misguided. And yes, it’s useful to regurgitate the typical “these benchmarks are flawed” and “it doesn’t represent the real world” (which is a pretty condescending thing to say to strangers, really, what people mean is it doesn’t represent THEIR real world).

I know that we’re largely saying the same thing, I just wanted to get the last word in :slight_smile:

1
Post #19
sasajuric

sasajuric

Author of Elixir In Action

That’s debatable. Last time I checked, these tests took only 10 sec, so they didn’t really measure the cost of garbage collection (which might significantly affect the results for stop-the-world-gc runtimes). They also didn’t observe how CPU bound tasks interfere with I/O ones (which under some conditions can cause tremendous latency increase). For other issues, see comments made by @cmkarlsson in this thread.

It’s not just about language. There are properties, such as fault-tolerance, the ability to troubleshoot production, ecosystem, and whatnot. Those things matter, and IMO matter much more than the speed. Not that speed is irrelevant, but past some point of “acceptable”, it matters much less, and sometimes it can be counterproductive.

There is a difference between performance matters and “I want to use every nanosecond of CPU time as best as possible”. Of course that all of us want to have some reasonable performance. On the other hand, aiming for the fastest possible framework (there’s no such thing, but let’s pretend it exists) is IMO usually wrong, b/c that speed gain is likely obtained by some trade-off, which might not be immediately obvious.

My usual advice is to measure whether the candidate framework is good enough for the desired case. That’s what I did when I first evaluated Erlang, by running a 12 hours load test on a simulation of the system, with 10x of the estimated load. Once I was convinced that Erlang easily handles that (and that it can easily scale), I didn’t really care about the speed anymore. I knew that if in some special cases I need to squeeze out the best performance, I can easily step outside of Erlang for that.

When I mention “real world” I talk about systems running in production (and those which are being developed to run in production in the future). If it’s running in production, it’s real. Otherwise it’s not. Consequently, TE benches are not real-world, but rather some (IMO very poor) attempt to simulate the production. If that makes me condescending, so be it :slight_smile:

The problem is that your production is not the same as mine, and definitely not the same as the thing being benched in TE. Hence, even though framework foo might be 100x faster than bar in the TE bench, it might well happen that bar actually produces better number for your case (or mine). Therefore, even if TE benches were done properly (and I don’t think they are), they still wouldn’t tell you a lot about your system, and might even lead you to a bad conclusion.

Which is why I believe there’s no such thing as the proper general bench comparison of frameworks. You can design one for your own system, simply to discard any framework which makes it very hard to deal with the desired load. Past that point, the decision on which framework to use should IMO mostly revolve around other properties.

5
Post #20
anthonyb

anthonyb

Yes, you can install and run the TechEmpower benchmarks suite locally via the Vagrant setup they provide. The basics are in the readme, with further details in the documentation.

Among fullstack frameworks using a full ORM (as opposed to raw SQL) with an RDBMS (i.e., MySQL or Postgres), on most of the tests, the performance of Phoenix is respectable, though not exceptional. However, performance on the “Data updates” and “Multiple queries” tests is especially low (with a significant regression on “Data updates” from Round 13 to Round 14) – on the lower end of the fullstack Python and even PHP frameworks. It might be worth investigating what’s going on with those two tests in particular. Is it something with Ecto?

More generally, it might not be a bad idea for the core team members to take a look at the Phoenix code submitted for these benchmarks and ensure Phoenix is putting its best foot forward. The Phoenix website describes Phoenix as “Productive. Reliable. Fast.”. Whatever flaws these benchmarks may have, people will likely look at them and conclude Phoenix isn’t living up to that third adjective.

Note, it isn’t difficult to push changes to the TechEmpower code. Just use their Vagrant setup to make changes on a local machine and send a pull request to the Github repo.

It might also be interesting to add another entry using just Cowboy, Plug, and raw SQL – this could help show the overhead added by Phoenix and Ecto.

4
Post #21