That’s exactly what we did during our testing - we parsed (with both poison and jiffy), giving us performance below 20,000 QPS. Like I said, our scala code is at 85,000 so obviously that would make Elixir an unsuitable replacement.
Maybe we’re only testing how fast we can open and close tcp acceptors but in our workload we need that to be capable of at least 100,000 QPS and from other peoples benchmarks (like, take Whatsapp for example) doing this should definitely be possible.
This reminds me: have you tried providing +K true to Erlang VM (enables kernel poll, default is false), and/or increasing the number of async threads for I/O with +A n (default is 10)?
Interesting I have tried myself to benchmark a simple plug on my Macbook. The Plug code is listed below and started by
{:ok, _} = Plug.Adapters.Cowboy.http SimplePlug, []
I get similar results to yourself I cannot get more than 26,000 Requests/sec. Tried setting +K true also.
defmodule SimplePlug do
import Plug.Conn
def init(options) do
options
end
def call(conn, _opts) do
conn
|> put_resp_content_type("text/plain")
|> send_resp(200, "Hello")
end
end
I tried a simple Java spark example.
import static spark.Spark.*;
public class Main {
public static void main(String[] args) {
get("/", (req, res) -> "Hello");
}
}
It gets about 35,000 Requests/sec. So it’s not that much ahead of Elixir.
Go out of the box, blows them all away…
package main
import (
"io"
"net/http"
)
func handler(writer http.ResponseWriter, request *http.Request) {
io.WriteString(writer, "Hello")
}
func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
This gets about 51,000 Requests/sec
Also did some runs looking at the processes under observer.
Using wrk -c 40 -d 180s -t 2 --latency http://localhost:4000/ to load up the plug. I don’t see any process’ queue growing longer then 3 msgs. I suspect on a Macbook with 4 cores about 25,000 Requests/sec is as good as it gets with Elixir/Plug.
Not that this is a bad figure, I did a very simple Python 3 async benchmark to see what it could do with the exact same wrk command. The python code was:
import asyncio
from aiohttp import web
async def handle(req):
return web.Response(body=b"Hello")
app = web.Application()
app.router.add_route('GET', '/', handle)
web.run_app(app)
Python3 only managed to get about 2,300 Requests/sec so Elixir/Plug is about 10 times faster than Python3 doing similar task under same load conditions.
That’s fair, but what’s weird is that these are still much lower numbers than what has been achieved before. I wonder if there was some regression in cowboy? Gonna need to do some digging.
I’ve tried benching SimplePlug on my MBP. I built an OTP release and ran the 3min test. Here’s what I got:
Running 3m test @ http://localhost:4000/
2 threads and 40 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 27.05ms 72.63ms 804.03ms 88.79%
Req/Sec 25.74k 8.24k 35.75k 86.90%
Latency Distribution
50% 705.00us
75% 1.09ms
90% 121.27ms
99% 312.24ms
8003062 requests in 3.00m, 1.41GB read
Requests/sec: 44441.19
Transfer/sec: 8.02MB
The latency leaves a lot to be desired, but the throughput is nice. I’ve also seen varying results on repeatable runs. For example, this one had better (but still not good) latency, but a lower throughput:
Running 3m test @ http://localhost:4000/
2 threads and 40 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.71ms 44.46ms 558.37ms 94.47%
Req/Sec 27.17k 6.56k 49.11k 90.48%
Latency Distribution
50% 644.00us
75% 0.90ms
90% 1.44ms
99% 242.17ms
7420593 requests in 3.00m, 1.31GB read
Socket errors: connect 0, read 1, write 0, timeout 0
Requests/sec: 41206.18
Transfer/sec: 7.43MB
For some shorter tests, I was able to get more stellar results:
Running 30s test @ http://localhost:4000/
2 threads and 40 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.52ms 8.52ms 169.09ms 98.74%
Req/Sec 28.17k 3.94k 33.88k 81.51%
Latency Distribution
50% 651.00us
75% 0.87ms
90% 1.18ms
99% 17.46ms
1638500 requests in 30.04s, 295.63MB read
Requests/sec: 54550.41
Transfer/sec: 9.84MB
I didn’t investigate this deeper. One possible reason is that this is my devbox, and there’s a bunch of stuff running on the system. It probably didn’t help that the load test was running on the same box as well. Also, since it’s OS X, I couldn’t play with +sbt flags which might make a difference.
Given the numbers I get locally on a devbox where a lot of other apps are running in the background, I’m fairly confident that better results can be observed on the dedicated server.
I spent some time last night “tinkering” with interesting results.
Cowboy 2.0, using a hacked up copy of their examples/post_data can do 106k/sec 204’ing without reading the body on the machines I struggled to get over 25k on with elixir/plug.
Elli could do 160k on the same hardware.
I tried to use one of the elli plug adapters and got 2k QPS through Elixir. The weirdest part of my Elli testing was I was using rebar3 shell and that creates +A 0 and +K false and I still got 160k. I am sure if I build a release and tune it properly I can drive that upwards of 250-300k.
Just figured I’d put a mini update as I’m not finished (very slowly) trying to untangle this performance issue. My next plan is cowboy 1.1, which is what Plug is using, to see if I reach a bottleneck there.
Then adding raw erlang body reading into both cowboy and elli to see what that looks like.
This is definitely suspicious. Phoenix in dev is going to do code reloading and other things. If you are getting the same performance in both, there is definitely something else at play.
I would expect the default Phoenix stack in production to be around a 10% impact on a pure Plug stack. Also remember that logging has a huge impact in performance, make sure both apps are logging about the same amount of information.
If results are still suspicious, please let us know how you installed Erlang and which flags were used during the installation.
The last part of your post caught my eye, @josevalim , (I haven’t managed to resolve the issue yet, the closest I’ve gotten is 160k using elli with raw erlang. I plan to try and tackle this again on the weekend).
I installed erlang both from “kerl” (on fedora23) and from the erlang-solutions package repo on Ubuntu. Are there additional optimizations of erlang that people don’t get out of the box that can be compiled in? I didn’t set any flags during the installation with kerl, just whatever the defaults were.