Which is the fastest web framework? (Link/Repo and results in this Topic)

I actually did that in a test I did after my post, but it gained less than 1% speed so I figured it was not worth it to post it. ^.^

1 Like

Haven’t checked it in years but I’d bet they set the checks to a TTL at some point to avoid the overhead.

1 Like

Your solution of passing arguments to mix release only affects the VM spawned to build the release - not the actual production release. You need to use the vm.args file when working with distillery as described here.

For example:

# in rel/config.exs
environment :prod do
  # ...
  set vm_args: "rel/vm.args"
end
# in rel/vm.args (we can use eex)
-name <%= release_name %>@127.0.0.1
+A 100
+K true
4 Likes

thanks, but not sure that is correct - it does affect the release vm.

doing a MIX_ENV=prod mix release --erl="+K false +A 1000" --no-tar

makes the erl_opts show up in the "$RELEASES_DIR/$REL_VSN/$REL_NAME.sh" script which is used to start the release vm.

# Options passed to erl
ERL_OPTS="${ERL_OPTS:-+K false +A 1000}"
# Environment variables for run_erl
RUN_ERL_ENV="${RUN_ERL_ENV:-+K false +A 1000}"

also this +A 1000 release will consistently yield much worse benchmarks - so very much seems the erl_opts are in effect.

https://hexdocs.pm/distillery/Mix.Tasks.Release.html#module-examples

# Pass args to erlexec when running the release
mix release --erl="-env TZ UTC"
4 Likes

Oh, I had no idea distillery does that - you’re right, I’m sorry. It doesn’t seem to be documented anywhere, though.

2 Likes

I love this community. Not only did y’all improve the reporting of benchmarks for Phoenix and Plug, but y’all also improved and added the benchmarks for other frameworks not even elixir related.

12 Likes

Can’t know how well Elixir performs if the other things are not performing up to par as well. ^.^

4 Likes

Looking at the project repos, why is the plug version using elixir 1.4 and the phoenix version using elixir 1.2? Or is that going to be updated when Phoenix 1.3 comes out?

2 Likes

they use the same elixir version.

elixir: "~> 1.2" and elixir: "~> 1.4" are “approximately greater than” requirements and in this case running elixir 1.4, 1.5, 1.6, 1.7, 1.8, or 1.9 would satisfy those two requirements.

Just didn’t know what environment the suite was running in or see where it was defined.

2 Likes

At https://github.com/tbrand/which_is_the_fastest there’s comparison of the web frameworks.

It can give you a high level idea. And that is, Phoenix and Plug are slow compared to, for instance, Ruby and its framework Roda. Ruby/Roda is faster. Or at least roughly more or less the same as Phoenix and Plug.

I thought Phoenix and Plug would very well beat ruby and python with a magnitude of several times. But that’s not the case.

Why are Phoenix and Plug so slow?

1 Like

This benchmark does not benchmark anything useful. There is no business logic involved, and if you need three simple routes that do virtually nothing, I see no point in using any framework at all.

6 Likes

Already discussed

2 Likes

Actually once they switched to wrk as a better benchmark tool, Elixir jumped to be faster than Ruby: https://github.com/tbrand/which_is_the_fastest/pull/223/files

So I would take any results there with a big pinch of salt. Don’t put too much weight onto silly benchmarks like this, instead try yourself with a realistic workload.

And remember: being the fastest in one workload means you make tradeoffs relative to other workloads, not to mention tradeoffs in programmer productivity. There are many other things than just raw single thread performance to optimise for when choosing a language. Work smarter, not harder.

8 Likes

I care about its being faster rather than me being more productive.

Faster under which circumstances?

Just sending out an empty response? Or under the load of actually doing work?

This benchmark does only measure delivering of staticaly empty content. This is probably never what one considers real workload.

And what is it you want the speed? TTFB? Requests per seconds? Calculating the full response body? Something else?

8 Likes

I am the one who added Plug and Phoenix to that project, out of curiosity. If I only knew this was the kind of discussion it would generate I would think a little bit more before doing it.

Just for instance, when I added, both Plug and Phoenix were only losing to Go and Crystal microframeworks. Phoenix, even being a full-featured framework, did beat express and some other “very fast” frameworks at that time. One thing that might have happened is that devs from other languages put loads of code and config to optimize it, while I didn’t take the project as seriously to spend so much time optimizing it, the Plug and Phoenix projects there are quite simple (I even suspect they still using Poison, which we know is a perfomance bottleneck of the default projects config).

Anyway, that is just a fun project, and as everyone mentioned, comparison benchmarks like this aren’t very reliable since they are not very good imitating real life situations. So don’t take it as an argument to use or not Elixir or whatever else you want to test. Just test it.

I used to think like that too. But then I realized I should be programming in C if I think like that, and made the choice to sacrifice a little bit of performance in favor of some happiness. :smile:

8 Likes

This is a brilliant example of how shallow and misleading these benchmarks can be.

This is one of the reasons why I think such benchmarks are useless. The results highly depends on how much effort is invested into each particular implementation. IMO, it makes the benchmark completely unscientific. Considering alo that, as others have mentioned, this benchmark measures nothing realistic, I find it completely useless.

Precisely! In some cases raw speed and every nanosecond matter, but then Ruby is not a good choice anyway. Something like C or e.g. Rust would IMO fit better in such circumstances.

6 Likes

Yes…

I used to think like that too.
For my case I care about performance and how much resources it eats.