Which is the fastest web framework? (Link/Repo and results in this Topic)

phoenix
benchmark
Tags: #<Tag:0x00007fbcb26fa010> #<Tag:0x00007fbcb26f9e80>

#21

Hmm, odd, plug is working fine, and I did grab the head of your PR, how odd…

The client is crystal too, it is not capable of saturating the elixir or rust servers I’m seeing based on my CPU and memory usage. This is not a good benchmarking tool.


#22

Is Phoenix kicking ass yet? :sunglasses:


#23

Phoenix has always been kicking butt :lol:

Let’s see how other frameworks/languages perform for things like 2 million WS connections :003:


#24

The PR looks like the build is failing because of…lack of a Rake file? Do the requirements say you need to provide a travis.yml because it looks like it’s defaulting to Ruby due to lack of one?

https://travis-ci.org/tbrand/which_is_the_fastest/builds/230391670


#25

I totally agree; it is able to saturate my quad core i7 for the elixir benchmarks though…

you can up the threads in the benchmarker '#{CLIENT} -t 16 -r 5000' - but all these keep_alive, do nothing, no garbage collection benchmarks are ¯_(ツ)_/¯


#26

I got this for results (such bad bad benchmarking… siege or any of those others would be far better to test with than this custom crystal client… plus it cannot even saturate the elixir server, blehg) for both plug and iron, since phoenix is not running yet, also added in express since @kelvinst was wanting to see how it compares, and I added in router.cr since it was the fastest on the chart, and I sorted my tests from fastest (top) to slowest:

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust                      iron                             4.559537        4.498832        4.534531
elixir                    plug                             5.444713        5.185388        5.326322
crystal                   router_cr                        9.139290        8.827565        9.007032
node                      express                         32.212435       31.668852       31.824345

Elixir is also not being warmed up yet either it seems, so again, inaccurate, but much of this benchmark is as about all benchmarks are. ^.^

Node also left its process running after its test was over, I had to kill it manually (prevented others from running)… >.<
But what do you expect from node, they don’t even die when their stdin dies, its like they always run in daemon mode, which is stupid…

Also, holy crap express took forever to benchmark (I’d been benchmarking it through my last couple of posts until now…).

Also, crystal is a lot slower here than on his chart, did he lie?

Well Plug is almost as fast as the native code languages like rust/crystal, it blew node entirely away.

Also @kelvinst, any reason phoenix is on 1.2 instead of 1.3rc?
Also @kelvinst, any reason the endpoint has a LOT of unused plugs in it?


The websocket should be removed.
The Plug.Static should be removed.
The Plug.RequestID should be removed.
The Plug.Logger should be removed.
The entire code reloading section could be removed.
The Plug.Parsers should be removed.
The Plug.MethodOverride should be removed.
The Plug.Head should be removed.
And in the router:

The plug :accepts should be removed.
The pipe_through :api should be removed.

You should only add things to Phoenix that are actually used, extra stuff for this test is not and he specifically asked in his readme I want to know the response time, not a usability., thus… phoenix is not quite right yet, a lot can be stripped out.


#27

it’s all about the cores… think the repo image is tested on dual core or something…

i7 quad:
Language (Runtime) Framework (Middleware) Max [sec] Min [sec] Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
elixir plug 5.751443 4.769079 5.290282
crystal router_cr 2.677777 2.580018 2.637836
node express 18.636735 17.242073 17.742885
crystal kemal 3.820253 3.596602 3.768468

they should also add cluster to express utilize more than one core.


#28

I tested on an utterly ancient Native 6-core AMD Phenom ][ at 3.6ghz on Ubuntu 17.04 freshly rebooted with not even an X GUI running on it at the time.

EDIT: @kelvinst Ooo saw your new push, looks nice! Let me see if I can grab and run it now. :slight_smile:


#29

Yes, I wanted to stay with a production release. Do we have any significant performance change in 1.3??

Due to generators, I guess we can cut off these ones for sure!

PR updated!


#30

Yeah the generators are for newbies primarily, I’ve not used a generator in phoenix in a while now… ^.^;

Absolutely not a clue, does anyone like @chrismccord know or so? :slight_smile:

Whooo I see, trying to update and see if it runs now and what score. :slight_smile:


#31

Added Phoenix latest pull above:

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust                      iron                             4.559537        4.498832        4.534531
elixir                    plug                             5.444713        5.185388        5.326322
elixir                    phoenix                          5.792088        5.673842        5.752746
crystal                   router_cr                        9.139290        8.827565        9.007032
node                      express                         32.212435       31.668852       31.824345

#32

Nice job fellas :023:


#33

added clustered express, to be fair. (killing the node servers is a big mess)

still quad i7

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
elixir                    plug                             5.751443        4.769079        5.290282
elixir                    phoenix                          7.742643        6.232111        6.921242
crystal                   router_cr                        2.677777        2.580018        2.637836
crystal                   kemal                            3.820253        3.596602        3.768468
rust                      iron                             3.889953        3.361527        3.669181
node                      express                         18.636735       17.242073       17.742885
node                      expresscluster                   8.418355        7.312338        7.875048

npm i cluster --save - app.js:

var cluster = require('cluster');
var express = require('express')
var numCPUs = require('os').cpus().length;


if (cluster.isMaster) {  
    for (var i = 0; i < numCPUs; i++) {
        // Create a worker
        cluster.fork();
    }
} else {
   var app = express()
   
   app.get('/', function (req, res) {
     res.send('')
   })
   
   app.get('/user/:id', function (req, res) {
     res.send(req.params.id)
   })
   
   app.post('/user', function (req, res) {
     res.send('')
   })
   
   app.listen(3000, function() {})
}

#34

Heh, so the node servers left running is not just an issue I had? Good to know. ^.^;

It was not responding to sighup, was not responding to sigquit, I eventually had to sigkill them, they just did not want to die when told to die. ^.^;


#35

That’s what we call resilience! Stubborn resilience to be more accurate! :lol:


#36

Lol, I call it a bug, or a virus. :stuck_out_tongue_winking_eye:


#37

Whooooa, that one did hurt! :lol:


#38

Lol, I don’t like services that don’t die when I sigquit them. And they should also die when they receive a sighup or they receive an EOL on <stdin> when running in non-daemon mode. :wink:


#39

found a potential fix:

benchmarker.cr:
in def kill move @process.not_nil!.kill to a last 'else’
and then replace the kill_proc("node") with
Process.run("pkill -9 -P #{@process.pid}", shell: true)

one should probably remove the entire kill_proc - its grep and kill - somewhat dangerous imho.

then make -B benchmarker

of course keep our elixir stop cmds - works for the nodejs, even in clustered…

  def kill
    #@process.not_nil!.kill

    # Since ruby's frameworks are running on puma, we have to kill the independent process
    if @target.name == "rails" ||
       @target.name == "roda" ||
       @target.name == "sinatra" ||
       @target.name == "express" ||
       @target.name == "expresscluster"
      Process.run("pkill -9 -P #{@process.pid}", shell: true)
    elsif @target.name == "plug"
      path = File.expand_path("../../../elixir/plug/_build/prod/rel/my_plug/bin/my_plug", __FILE__)
      Process.run("bash #{path} stop", shell: true)
      @process.not_nil!.kill
    else
      @process.not_nil!.kill
    end
  end

#40

I ran some more, still sorted from fastest to slowest as top to bottom:

Language (Runtime)        Framework (Middleware)          Max [sec]       Min [sec]       Ave [sec]
------------------------- ------------------------- --------------- --------------- ---------------
rust                      iron                             4.559537        4.498832        4.534531
rust                      nickel                           5.253805        5.165856        5.207764
elixir                    plug                             5.444713        5.185388        5.326322
elixir                    phoenix                          5.792088        5.673842        5.752746
rust                      rocket                           5.977488        5.849199        5.914374
crystal                   router_cr                        9.139290        8.827565        9.007032
crystal                   kemal                           10.316980        9.544119       10.050472
node                      express                         32.212435       31.668852       31.824345
ruby                      roda                            66.817984       63.877961       65.617340
ruby                      sinatra                        147.016259      144.732534      146.124146
ruby                      rails                          477.844824      475.863197      476.804092

I could not make go because:

╰─➤  make go
go get -u github.com/labstack/echo
package context: unrecognized import path "context" (import path does not begin with hostname)
Makefile:60: recipe for target 'echo' failed
make: *** [echo] Error 1

I could not make swift because:

╰─➤  make swift                                                                                                     1 ↵ cd swift/vapor; swift build --configuration release
error: unsatisfiable
Makefile:103: recipe for target 'vapor' failed
make: *** [vapor] Error 1

I’m getting some very different results from the chart on the repo, I’m wondering if my higher amount of (though slower) cores let’s the concurrent ones work better…

Also holy crap the ruby ones took sooooooo-veeeeeeeery-loooooooong:scream:
That was painful…
Ruby was the sole reason it took sooooooo long to put up this next set of tests…
I’m amazed at how slow ruby is, why would anyone use it?! I bet PHP is faster…

Hmm, not just plug but everything elixir/erlang should use the ‘stop’ command…