A typical bad benchmark, it just tests more or less bad ports of an algorithm.
One might even think that there is some intended outcome coded into the ports … even ruby could be faster than ASM … if you try hard enough to get a real bad port to ASM.
If one would realy try to make a benchmark fast, he would use all tricks the language provides … why on earth should an implementation in C++ be ever slower than in C ? They may even be identical, if C++ does not help to make it faster …
2 Likes
Yeah this is something I’ve mentioned a lot. There is absolutely no reason for any C++ code to be slower than any C code, at worst it will be equal, at best C++ will be able to generate MUCH better code mostly due to templates.
1 Like
I just discovered the TechEmpower benchmark , as I was considering a new (for me) framework / langauge for a personal project . Basically debating between sticking with Ruby ( and Rails or Padrino), or trying our Elixir / Phoenix or Crystal / Amber.
The benchmark empressed me enough that I wanted to see how well it would perform for my one of my use cases: large number of requests to a JSON endpoint that pulls a (random) row or a set of 10 from MariaDB table of half a million rows.
My conclusion - real life application is way different from a benchmark.
Also for my case I needed a full framework not a micro framework ( hence not Sinatra or similar). Because of the number of different features I need that I would need to implement - I would end up with a fairly large app anyways, so might as well save some time.
I ran wrk2 against all setups .
So bottom line - Amber was only about 3x times faster than Rails 5.2 . Padrino performed about the same as Rails. Phoenix was about in about the middle of the pack. Once I added a bit more juice to my test VM, Phoenix actually was able to scale up much better and overtake completion.
So my conclusion here is that while Rails is slowest of the bunch - it’s well within my requirements of being able to handle ~200 req per second under 100ms response times on a low-end VM on digital ocean or vultr.
However as I was going through these benchmarks I realized that I will need a lot of libraries that I’m dont know if Crystal / Elixir have, and I know very well how to optimize hosting for Rails, but it took me a bit of time to do it for Elixir and Amber ( and chances are I still didn’t do it right, since I’m still learning ). Also it has a massive advantage of a huge number of gems that’s will save me time, as a one-man side project (Crystal and Elixir also have a growing ecosystem, but’s it not there yet, it seems?)
I I’m still considering building out an simple api in Phoenix or Amber just for the fun of learning either one , and building Rails/Elm frontend for it .
My point here is that those benchmarks are not very useful, because they don’t take into consideration very important real life considerations, requirements and constrains
4 Likes
This is exactly how one should go about answering the question of “What should I use?”: Well done!
With regards to Crystal I think it seems like a much more reasonable way going forward than trying to make Ruby fast. I do think that they need a clear way to move entire gems into that performance space, though, and I’m not sure (from my fairly basic research) this is something they actually are aiming to do. It seems like a mistake to try to explicitly fix a bunch of Ruby problems and try to explicitly be a “fast Ruby” but then not have an efficient transition story for code.
4 Likes
Thank you for the compliment. I really want to spend some time learning either Phoenix or Amber, but I only have 2 months for this project. So I’m still debating which way to go ( till tomorrow). But that’s not something anybody but me can answer 
Conversely, when I was doing my tests, I was actually able to move most of my model/ migration/controller code with very little modification from Rails to Amber, just needed to add types to variables and several functions. Which was a pleasant surprise I might say? Of course, that was a trivial part of the app, which is kind of designed to be rails-esque, activerecord-ish.
Wake me up when they’ve cottoned on to the importance of immutability in Crystal++…
2 Likes
I’m obviously biased in my opinions, so please keep that in mind.
But for me, one of the biggest points after making sure performance is “good enough”, is looking at the backwards-compatibility story. And I will argue, Elixir has the best story. Elixir and Phoenix both had their first stable release about 3 years ago (Elixir in May of 2015, Phoenix in August). During that time they both kept almost perfect backwards compatibility (modulo some bug fixes and security issues).
Crystal was supposed to reach 1.0 in 2017, but from the looks of it, it’s still very far away from this goal even now. Breaking changes are relatively frequent (less so recently, though).
Rails is generally introducing breaking changes in their every x.1 release - this means 2 times in the last 3 years, with the next release which is not backwards-compatible right around the corner. Ruby is also not adhering strictly to SemVer and introduced couple minor breaking changes in the point releases.
From my experience with Rails and Ruby, every version bump of Rails or Ruby is a huge endeavour - you never know what might happen and you need to test extensively. On the other hand, I never had any issue upgrading either Elixir, Erlang or Phoenix version. It’s a huge advantage not many people talk about.
11 Likes
In other fields maybe, since Crystal is a bare metal language. But I believe as for Web framework, though is not fastest, Phoenix is superior compared to most of the others because distributing and fault tolerance (BEAM’s nature). And these two factors are very important for web servers, as well as very hard for programmer to make them right if it’s not under the infrastructure.
Elixir has a very important advantage no Oracle IP.
First of all, Crystal is NOT, absolutely NOT parallel. It is concurrent, but not parallel. In other words it is single-core only, you will never ever get use of more that one core on a system without sharding it out like nodejs does.
So at it’s basic setup, Phoenix will run circles around Amber on multi-core systems, however Amber will be faster on single-core systems.
Rails is absolutely abysmal in my tests, summary of just raw throughput:
Ranking by Average Requests per second:
- 680446 req/sec : bin/server_cpp_evhtp
- 518792 req/sec : bin/server_nim_mofuw
- 498347 req/sec : bin/server_go_fasthttprouter
- 368537 req/sec : bin/server_rust_iron
- 337937 req/sec : bin/server_rust_nickel
- 295340 req/sec : bin/server_go_iris
- 294538 req/sec : bin/server_rust_rocket
- 261883 req/sec : bin/server_go_echo
- 241104 req/sec : bin/server_go_gorilla_mux
- 196827 req/sec : bin/server_go_gin
- 195051 req/sec : bin/server_node_clusterpolka
- 183457 req/sec : bin/server_python_sanic
- 127754 req/sec : bin/server_elixir_plug
- 119078 req/sec : bin/server_elixir_phoenix
- 111798 req/sec : bin/server_csharp_aspnetcore
- 108408 req/sec : bin/server_python_japronto
- 100829 req/sec : bin/server_node_clusterexpress
- 65878 req/sec : bin/server_crystal_router_cr
- 65102 req/sec : bin/server_crystal_raze
- 52676 req/sec : bin/server_crystal_lucky
- 48605 req/sec : bin/server_crystal_kemal
- 31704 req/sec : bin/server_nim_jester
- 23244 req/sec : bin/server_node_polka
- 11100 req/sec : bin/server_node_express
- 5661 req/sec : bin/server_ruby_roda
- 4411 req/sec : bin/server_ruby_rack-routing
- 2034 req/sec : bin/server_ruby_sinatra
- 1783 req/sec : bin/server_python_flask.py
- 722 req/sec : bin/server_python_flask
- 594 req/sec : bin/server_ruby_rails
(Amber is not in the above results because it’s code is bugged but it’s close to the other crystal frameworks).
Amber is a LOT faster than Rails on raw throughput, but as always the user code will affect a lot.
Yep, this is because Crystal the language (and thus no framework on it) is not multi-core capable.
Eh, I don’t know about Crystal, but Elixir is not a one-man project, it is backed by a company and has a lot of authors at this point (with a BDFL though), in addition, as you can see on recent stack overflow benchmarks, Elixir is one of the fastest growing languages out and already in the top 20 of server usage.
Exactly, benchmarks are not useful for showing real information at all. I think micro-benchmarks (like my above linked raw throughput testing) can be useful to show the implicit overhead, but large benchmarks are near-useless.
You can get things built yourself fast in phoenix though, while having a far far faster base speed, and being significantly easier to reason about.
Yeah, I really really don’t like the base syntax of Crystal… Even Rust does default-immutable… >.>
Do not underestimate the importance of this!!!
‘Mostly’. There’s a reason I don’t call OCaml a bare-metal-language even though it’s near C speed and compiles to direct machine code, and Crystal has the same reason, it has a GC, and unlike OCaml’s GC, Crystal’s GC is the C’based HB Collector, really basic and not terribly fast, I.E. you get ‘non-base-metal’ random costs because of collections. In addition the GC only manages memory resource, not other resources, where if the language was better designed and could manage all resources in a scoped manner (like Rust does) then it wouldn’t need a GC and it would have a unified syntax for all resources. As it stands you either have to pass callbacks (that do get inlined admittedly) and do all the work inside that (think the with command in Python) or you have to explicitly and manually call the reclamation functions for the resource (and don’t forget that or it ‘might’ get reclaimed by the finalizer if a finalizer was made for it, and even then that will only be ‘sometime later’).
Phoenix also naturally scales across machines as well.
Crystal has Oracle IP?! o.O
3 Likes
Dah for some reason thought it compiles to JVM 
1 Like
I agree with @OvermindDL1 Elixir/Phoenix r than Crystal/Amber on multi-core, can’t compete with that at the moment, even though it is possible to use Amber in cluster mode in linux VMs.
Crystal and Amber has a lot to do to catch up in many things. For instance one of the things that makes Amber framework even slower is the current available DB drivers for the crystal language could be improved.
Benchmarks, is just a point of reference that should be considered but not taken too seriously in IMO.
Anyways I love Ruby, Elixir and Crystal they all great languages for developer happiness
