Hmm, odd, plug is working fine, and I did grab the head of your PR, how odd…
The client is crystal too, it is not capable of saturating the elixir or rust servers I’m seeing based on my CPU and memory usage. This is not a good benchmarking tool.
Is Phoenix kicking ass yet? 
Phoenix has always been kicking butt 
Let’s see how other frameworks/languages perform for things like 2 million WS connections 
3 Likes
The PR looks like the build is failing because of…lack of a Rake file? Do the requirements say you need to provide a travis.yml because it looks like it’s defaulting to Ruby due to lack of one?
https://travis-ci.org/tbrand/which_is_the_fastest/builds/230391670
I totally agree; it is able to saturate my quad core i7 for the elixir benchmarks though…
you can up the threads in the benchmarker '#{CLIENT} -t 16 -r 5000' - but all these keep_alive, do nothing, no garbage collection benchmarks are ¯_(ツ)_/¯
I got this for results (such bad bad benchmarking… siege or any of those others would be far better to test with than this custom crystal client… plus it cannot even saturate the elixir server, blehg) for both plug and iron, since phoenix is not running yet, also added in express since @kelvinst was wanting to see how it compares, and I added in router.cr since it was the fastest on the chart, and I sorted my tests from fastest (top) to slowest:
Language (Runtime) Framework (Middleware) Max [sec] Min [sec] Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust iron 4.559537 4.498832 4.534531
elixir plug 5.444713 5.185388 5.326322
crystal router_cr 9.139290 8.827565 9.007032
node express 32.212435 31.668852 31.824345
Elixir is also not being warmed up yet either it seems, so again, inaccurate, but much of this benchmark is as about all benchmarks are. ^.^
Node also left its process running after its test was over, I had to kill it manually (prevented others from running)… >.<
But what do you expect from node, they don’t even die when their stdin dies, its like they always run in daemon mode, which is stupid…
Also, holy crap express took forever to benchmark (I’d been benchmarking it through my last couple of posts until now…).
Also, crystal is a lot slower here than on his chart, did he lie?
Well Plug is almost as fast as the native code languages like rust/crystal, it blew node entirely away.
Also @kelvinst, any reason phoenix is on 1.2 instead of 1.3rc?
Also @kelvinst, any reason the endpoint has a LOT of unused plugs in it?
https://github.com/tbrand/which_is_the_fastest/blob/master/elixir/phoenix/lib/my_phoenix/endpoint.ex
The websocket should be removed.
The Plug.Static should be removed.
The Plug.RequestID should be removed.
The Plug.Logger should be removed.
The entire code reloading section could be removed.
The Plug.Parsers should be removed.
The Plug.MethodOverride should be removed.
The Plug.Head should be removed.
And in the router:
https://github.com/tbrand/which_is_the_fastest/blob/master/elixir/phoenix/web/router.ex
The plug :accepts should be removed.
The pipe_through :api should be removed.
You should only add things to Phoenix that are actually used, extra stuff for this test is not and he specifically asked in his readme I want to know the response time, not a usability., thus… phoenix is not quite right yet, a lot can be stripped out.
1 Like
it’s all about the cores… think the repo image is tested on dual core or something…
i7 quad:
Language (Runtime) Framework (Middleware) Max [sec] Min [sec] Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
elixir plug 5.751443 4.769079 5.290282
crystal router_cr 2.677777 2.580018 2.637836
node express 18.636735 17.242073 17.742885
crystal kemal 3.820253 3.596602 3.768468
they should also add cluster to express utilize more than one core.
1 Like
I tested on an utterly ancient Native 6-core AMD Phenom ][ at 3.6ghz on Ubuntu 17.04 freshly rebooted with not even an X GUI running on it at the time.
EDIT: @kelvinst Ooo saw your new push, looks nice! Let me see if I can grab and run it now. 
Yes, I wanted to stay with a production release. Do we have any significant performance change in 1.3??
Due to generators, I guess we can cut off these ones for sure!
PR updated!
Yeah the generators are for newbies primarily, I’ve not used a generator in phoenix in a while now… ^.^;
Absolutely not a clue, does anyone like @chrismccord know or so?
Whooo I see, trying to update and see if it runs now and what score.
1 Like
Added Phoenix latest pull above:
Language (Runtime) Framework (Middleware) Max [sec] Min [sec] Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust iron 4.559537 4.498832 4.534531
elixir plug 5.444713 5.185388 5.326322
elixir phoenix 5.792088 5.673842 5.752746
crystal router_cr 9.139290 8.827565 9.007032
node express 32.212435 31.668852 31.824345
3 Likes
added clustered express, to be fair. (killing the node servers is a big mess)
still quad i7
Language (Runtime) Framework (Middleware) Max [sec] Min [sec] Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
elixir plug 5.751443 4.769079 5.290282
elixir phoenix 7.742643 6.232111 6.921242
crystal router_cr 2.677777 2.580018 2.637836
crystal kemal 3.820253 3.596602 3.768468
rust iron 3.889953 3.361527 3.669181
node express 18.636735 17.242073 17.742885
node expresscluster 8.418355 7.312338 7.875048
npm i cluster --save - app.js:
var cluster = require('cluster');
var express = require('express')
var numCPUs = require('os').cpus().length;
if (cluster.isMaster) {
for (var i = 0; i < numCPUs; i++) {
// Create a worker
cluster.fork();
}
} else {
var app = express()
app.get('/', function (req, res) {
res.send('')
})
app.get('/user/:id', function (req, res) {
res.send(req.params.id)
})
app.post('/user', function (req, res) {
res.send('')
})
app.listen(3000, function() {})
}
2 Likes
Heh, so the node servers left running is not just an issue I had? Good to know. ^.^;
It was not responding to sighup, was not responding to sigquit, I eventually had to sigkill them, they just did not want to die when told to die. ^.^;
2 Likes
That’s what we call resilience! Stubborn resilience to be more accurate!
Lol, I call it a bug, or a virus. 
2 Likes
Whooooa, that one did hurt!
Lol, I don’t like services that don’t die when I sigquit them. And they should also die when they receive a sighup or they receive an EOL on <stdin> when running in non-daemon mode. 
1 Like
found a potential fix:
benchmarker.cr:
in def kill move @process.not_nil!.kill to a last ‘else’
and then replace the kill_proc("node") with
Process.run("pkill -9 -P #{@process.pid}", shell: true)
one should probably remove the entire kill_proc - its grep and kill - somewhat dangerous imho.
then make -B benchmarker
of course keep our elixir stop cmds - works for the nodejs, even in clustered…
def kill
#@process.not_nil!.kill
# Since ruby's frameworks are running on puma, we have to kill the independent process
if @target.name == "rails" ||
@target.name == "roda" ||
@target.name == "sinatra" ||
@target.name == "express" ||
@target.name == "expresscluster"
Process.run("pkill -9 -P #{@process.pid}", shell: true)
elsif @target.name == "plug"
path = File.expand_path("../../../elixir/plug/_build/prod/rel/my_plug/bin/my_plug", __FILE__)
Process.run("bash #{path} stop", shell: true)
@process.not_nil!.kill
else
@process.not_nil!.kill
end
endI ran some more, still sorted from fastest to slowest as top to bottom:
Language (Runtime) Framework (Middleware) Max [sec] Min [sec] Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust iron 4.559537 4.498832 4.534531
rust nickel 5.253805 5.165856 5.207764
elixir plug 5.444713 5.185388 5.326322
elixir phoenix 5.792088 5.673842 5.752746
rust rocket 5.977488 5.849199 5.914374
crystal router_cr 9.139290 8.827565 9.007032
crystal kemal 10.316980 9.544119 10.050472
node express 32.212435 31.668852 31.824345
ruby roda 66.817984 63.877961 65.617340
ruby sinatra 147.016259 144.732534 146.124146
ruby rails 477.844824 475.863197 476.804092
I could not make go because:
╰─➤ make go
go get -u github.com/labstack/echo
package context: unrecognized import path "context" (import path does not begin with hostname)
Makefile:60: recipe for target 'echo' failed
make: *** [echo] Error 1
I could not make swift because:
╰─➤ make swift 1 ↵ cd swift/vapor; swift build --configuration release
error: unsatisfiable
Makefile:103: recipe for target 'vapor' failed
make: *** [vapor] Error 1
I’m getting some very different results from the chart on the repo, I’m wondering if my higher amount of (though slower) cores let’s the concurrent ones work better…
Also holy crap the ruby ones took sooooooo-veeeeeeeery-loooooooong… 
That was painful…
Ruby was the sole reason it took sooooooo long to put up this next set of tests…
I’m amazed at how slow ruby is, why would anyone use it?! I bet PHP is faster…
Hmm, not just plug but everything elixir/erlang should use the ‘stop’ command…
1 Like
I made a PHP file of (this is to follow his requirements in the README.md precisely):
<?php
$uri = $_SERVER['REQUEST_URI'];
if($uri == "/") die("");
else if($uri == "/user") die("");
else if(substr($uri, 0, 6) == "/user/") die(substr($uri, 6));
?>
And tested with that using the hhvm php server since that is the only one I have installed at the moment, added the results to the rest here:
Language (Runtime) Framework (Middleware) Max [sec] Min [sec] Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust iron 4.559537 4.498832 4.534531
rust nickel 5.253805 5.165856 5.207764
elixir plug 5.444713 5.185388 5.326322
elixir phoenix 5.792088 5.673842 5.752746
rust rocket 5.977488 5.849199 5.914374
crystal router_cr 9.139290 8.827565 9.007032
crystal kemal 10.316980 9.544119 10.050472
php hhvm 18.855467 18.748603 18.801522
node express 32.212435 31.668852 31.824345
ruby roda 66.817984 63.877961 65.617340
ruby sinatra 147.016259 144.732534 146.124146
ruby rails 477.844824 475.863197 476.804092
I’m impressed, PHP is pretty dang fast, especially considering it has to hit the filesystem to check if the file’s been changed on every request. PHP utterly blows Ruby away in speed, but still does not beat crystal/elixir/rust…
3 Likes
