Will Amber Framework (Crystal) perform better than Phoenix?

[OvermindDL1]

@konung

I just discovered the TechEmpower benchmark , as I was considering a new (for me) framework / langauge for a personal project . Basically debating between sticking with Ruby ( and Rails or Padrino), or trying our Elixir / Phoenix or Crystal / Amber.

First of all, Crystal is NOT, absolutely NOT parallel. It is concurrent, but not parallel. In other words it is single-core only, you will never ever get use of more that one core on a system without sharding it out like nodejs does.

So at it’s basic setup, Phoenix will run circles around Amber on multi-core systems, however Amber will be faster on single-core systems.

@konung

So bottom line - Amber was only about 3x times faster than Rails 5.2 . Padrino performed about the same as Rails. Phoenix was about in about the middle of the pack. Once I added a bit more juice to my test VM, Phoenix actually was able to scale up much better and overtake completion.

Rails is absolutely abysmal in my tests, summary of just raw throughput:

Ranking by Average Requests per second:

1. 680446 req/sec : bin/server_cpp_evhtp
2. 518792 req/sec : bin/server_nim_mofuw
3. 498347 req/sec : bin/server_go_fasthttprouter
4. 368537 req/sec : bin/server_rust_iron
5. 337937 req/sec : bin/server_rust_nickel
6. 295340 req/sec : bin/server_go_iris
7. 294538 req/sec : bin/server_rust_rocket
8. 261883 req/sec : bin/server_go_echo
9. 241104 req/sec : bin/server_go_gorilla_mux
10. 196827 req/sec : bin/server_go_gin
11. 195051 req/sec : bin/server_node_clusterpolka
12. 183457 req/sec : bin/server_python_sanic
13. 127754 req/sec : bin/server_elixir_plug
14. 119078 req/sec : bin/server_elixir_phoenix
15. 111798 req/sec : bin/server_csharp_aspnetcore
16. 108408 req/sec : bin/server_python_japronto
17. 100829 req/sec : bin/server_node_clusterexpress
18. 65878 req/sec : bin/server_crystal_router_cr
19. 65102 req/sec : bin/server_crystal_raze
20. 52676 req/sec : bin/server_crystal_lucky
21. 48605 req/sec : bin/server_crystal_kemal
22. 31704 req/sec : bin/server_nim_jester
23. 23244 req/sec : bin/server_node_polka
24. 11100 req/sec : bin/server_node_express
25. 5661 req/sec : bin/server_ruby_roda
26. 4411 req/sec : bin/server_ruby_rack-routing
27. 2034 req/sec : bin/server_ruby_sinatra
28. 1783 req/sec : bin/server_python_flask.py
29. 722 req/sec : bin/server_python_flask
30. 594 req/sec : bin/server_ruby_rails

(Amber is not in the above results because it’s code is bugged but it’s close to the other crystal frameworks).

Amber is a LOT faster than Rails on raw throughput, but as always the user code will affect a lot.

@konung

Once I added a bit more juice to my test VM, Phoenix actually was able to scale up much better and overtake completion.

Yep, this is because Crystal the language (and thus no framework on it) is not multi-core capable.

@konung

(Crystal and Elixir also have a growing ecosystem, but’s it not there yet, it seems?)

Eh, I don’t know about Crystal, but Elixir is not a one-man project, it is backed by a company and has a lot of authors at this point (with a BDFL though), in addition, as you can see on recent stack overflow benchmarks, Elixir is one of the fastest growing languages out and already in the top 20 of server usage.

@konung

My point here is that those benchmarks are not very useful, because they don’t take into consideration very important real life considerations, requirements and constrains

Exactly, benchmarks are not useful for showing real information at all. I think micro-benchmarks (like my above linked raw throughput testing) can be useful to show the implicit overhead, but large benchmarks are near-useless.

@konung

Thank you for the compliment. I really want to spend some time learning either Phoenix or Amber, but I only have 2 months for this project. So I’m still debating which way to go ( till tomorrow). But that’s not something anybody but me can answer

You can get things built yourself fast in phoenix though, while having a far far faster base speed, and being significantly easier to reason about.

@mmport80

Wake me up when they’ve cottoned on to the importance of immutability in Crystal++…

Yeah, I really really don’t like the base syntax of Crystal… Even Rust does default-immutable… >.>

@michalmuskala

But for me, one of the biggest points after making sure performance is “good enough”, is looking at the backwards-compatibility story. And I will argue, Elixir has the best story. Elixir and Phoenix both had their first stable release about 3 years ago (Elixir in May of 2015, Phoenix in August). During that time they both kept almost perfect backwards compatibility (modulo some bug fixes and security issues).

Do not underestimate the importance of this!!!

@namelos

since Crystal is a bare metal language.

‘Mostly’. There’s a reason I don’t call OCaml a bare-metal-language even though it’s near C speed and compiles to direct machine code, and Crystal has the same reason, it has a GC, and unlike OCaml’s GC, Crystal’s GC is the C’based HB Collector, really basic and not terribly fast, I.E. you get ‘non-base-metal’ random costs because of collections. In addition the GC only manages memory resource, not other resources, where if the language was better designed and could manage all resources in a scoped manner (like Rust does) then it wouldn’t need a GC and it would have a unified syntax for all resources. As it stands you either have to pass callbacks (that do get inlined admittedly) and do all the work inside that (think the with command in Python) or you have to explicitly and manually call the reclamation functions for the resource (and don’t forget that or it ‘might’ get reclaimed by the finalizer if a finalizer was made for it, and even then that will only be ‘sometime later’).

@namelos

Phoenix is superior compared to most of the others because distributing and fault tolerance (BEAM’s nature).

Phoenix also naturally scales across machines as well.

@andre1sk

Elixir has a very important advantage no Oracle IP.

Crystal has Oracle IP?! o.O