Plug POST performance

First thing i noticed is that you’re calling wrk on ‘/supply-partners/mopuba’ where your icepick code has the route defined as ‘/supply-partners/mopub’ (without the trailing ‘a’) this could give you a different code handling path than you intended. At least something to try

Have you seen this benchmark? GitHub - mroth/phoenix-showdown: 🏇 benchmark Sinatra-like web frameworks · GitHub
This one has already way higher request rates you’re getting now, maybe try that one first on the same hardware to see how it performs and use it as a baseline?

Another thing you could try is compiling with HiPe (high performance erlang) compiler through usage of ERL_COMPILER_OPTIONS="[native,{hipe, [o3]}]" when calling mix compile.

You could also try using one of the (erlang) profiling tools to see which functions take most of your CPU cycles.

One last thing i noticed when testing your stuff out on a simple macbook air 11" is that i could double the req/sec when i changed the json payload from the one above to just ‘{}’. So it seems it could have something to do with string handling of the incoming json. On the macbook air 11" i got 11k req/sec.

Goodluck!

Oh, we definitely tested with the right URL, sorry, just a bad paste of one of the many we tried (we were just testing fallback vs actually correct route at that time).

I’ll play around with the phoenix code (we actually tried, at one point, a new phoenix app modified to just 204, same results) as well as HiPE to see what that does. I played around with eflame for profiling but all it really showed me was most of the time was spent in cowboy/ranch. I did a couple of graphs by "grep -v"ing out the cowboy code and very little was left.

I’d be curious and surprised if you saw the same results using the “Ebid” code I had in the gist, as it shouldn’t really pay any attention to the body contents. From what I can see by reading plug code that only kicks in if you are actually using one of the parsers, and that code doesn’t parse anything.

Maybe I’m missing something, but the code in the repo seems to do more than that. If I simplify that top-level plug, to just return 204:

defmodule Icepick.PlugRouter do
  import Plug.Conn

  def init(options) do
    options
  end

  def call(conn, _opts) do
    conn
    |> send_resp(204, [])
  end

  def start_link() do
    {:ok, _} = Plug.Adapters.Cowboy.http(
      __MODULE__,
      [],
      [port: 8000])
  end
end

I get the following on my dev machine:

$ wrk -c 100 -d 60 -t 2 -s wrk.lua  http://localhost:8000/supply-partners/mopub
Running 1m test @ http://localhost:8000/supply-partners/mopub
  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    26.40ms   79.89ms 291.43ms   91.53%
    Req/Sec    22.38k     8.47k   46.78k    79.63%
  2510775 requests in 1.00m, 362.02MB read
Requests/sec:  41846.51

The latency here is still unusually large, but at least the throughput is way better (originally it was below 20k/sec).

Recently I demoed a simple Phoenix app which did some parameter decoding and json encoding (so a bit more work than this simple example), and I got a sub-micromillisecond 99th percentile latency with a wrk test (this same machine), so I’m pretty certain there’s room for improvements here.

That’s effectively what we had in the second link in the top post:

Which we also had trouble getting good performance with. It was far below the ~300k we see with scala. Definitely better throughput with that code in your testing. How powerful a dev machine (I’m guessing just a laptop?).

I’m VERY interested in the fact you had sub-millisecond with Phoenix. One of the worst issues we saw in our testing was latency of over 9 seconds. The machine was 60% idle from a CPU perspective at the time and we were completely baffled by how that could happen.

Yeah, it’s a 2013 MacBook pro, quad core 2.6GHz, 16MB RAM.

I expect scala will usually perform better on such synthetic small benchmarks. I wonder though how would it look when it’s a more real-life-like scenario, where these plugs do some real work both CPU and I/O bound, and they allocate some memory which needs to be GC-ed at some point.

Also a question: in your Elixir Plug bench, was CPU fully utilised under load?

If you start with a generated Phoenix app, there might be some issues. First, make sure to run the system as an OTP release (using exrm is best). Another important tip is to either increase log level to :warn or decrease Plug.Logger level to :debug. Otherwise, all requests are logged, and there’s a huge amount of requests under load-test. IIRC, Logger will start applying back-pressure when the queue gets bigger, and that might cause the bottleneck, and CPU under-utilisation during increased load.

Other common mistake is running JSON requests through the browser pipeline (use the API pipeline). Finally, if you’re testing locally, don’t use too many threads, because wrk might interfere with the VM.

Just to be clear, my test was very shallow. I had a simple endpoint with one browser req, and one API req, and issued them with 1:1 ratio with wrk for 30 seconds. That got me about 30k reqs/sec and sub-ms times in 99% percentile.

We do 85,000 QPS in Scala with a full workload (parsing the json, and making decisions based on that) and 300,000 QPS when we just 204. That’s why I’m so frustrated, I can’t get the performance anywhere even similar in the simple case to our more complex case.

In no case was the CPU ever fully loaded, we were averaging about 60% idle which is why I became convinced we must have hit some “bottleneck”, but we couldn’t prove that out.

We definitely didn’t change the logging level or use exrm so I’ll start from there and see where we can go from there. We’re definitely doing something silly as other people report over 200k QPS for similar tests on Elixir.

Until you’re able to get to 100% CPU usage under the load you’re probably hitting some bottleneck somewhere. Here are some additional suggestions for discovering it:

• Whether you’re using exrm or not, make sure to load test the code compiled with MIX_ENV=prod
• If you’re load testing simple 204 response, make sure to keep Plug.Parsers out. Using it might cause the body to be loaded (see Plug.Parsers.JSON), and the throughput of your simple CPU-bound code might become I/O bound.
• Consider temporarily moving Plug/http out of the picture. Write a simple program that spawns a couple of processes (at least as many as you have cores), and make those processes CPU bound. They could run an infinite loop which invokes :rand.uniform, or something similar. Just make sure they’re not receiving messages, sleeping, or doing I/O bound work. Check if you’re getting 100% CPU usage. If not, then take try playing with erl options and see if it helps.
• Start the observer, then load test the system for a while. If you have a bottleneck in the system, it might show up in the processes tab during the load test. Sort the view by reds and/or message queue, and try to see if some process is constantly on/near the top of the list with a large message queue. If yes, that’s a possible bottleneck.

Only when you reach CPU usage of 100% can you consider comparing Scala vs Erlang/Elixir, and also think about possible optimizations in your code. Before that, some other bottleneck is likely making your system less efficient than it could be.

Good luck!

Just a random tough. But did you compiled it with a prod mix environement ? That can have massive impact sometimes.

Yes, we definitely compiled with prod, we didn’t see a huge difference between dev/prod because dev consolidates protocols these days.

by not doing anything in the request body you’re just measuring how fast the language can open and close tcp acceptors. try doing something like unserializing and serializing a json body to the requests and benchmark again