Plug POST performance

apenney

At work we have a lot of Scala code in our low latency/high throughput product and a couple of us have been advocating Elixir as an alternative to the current codebase. We recently undertook a 24 hour hackathon and tried to replace a chunk of our code with Elixir but immediately ran into performance issues. I’m hoping other people on the list might have some suggestions for me because I’ve picked the brains of the IRC channel and been unable to improve things.

To begin with I’ll set the stage for the testing we did. We wanted to start by understanding the raw performance of Elixir “out of the box”, and so we whipped up a couple of codebases to accept POSTs and return 204 regardless of what happens.

We used this code:

https://github.com/apenney/icepick

Alongside a less complex piece of code:

We used wrk and wrk2 to performance various POST tests against this code. We used the following script to feed wrk with data to post:

For an example of the flags we used with wrk:

We did the testing on DigitalOcean machines as well as machines inside our own datacenter. We were unable to get beyond 22,000 QPS with any of our testing, no matter what we tried tweaking. On the same hardware we can swap in some scala and do over 300,000 QPS easily.

Things that we tried:

Tweaking sysctls.
Running wrk from the same box to rule out the network.
GET instead of POST (45k QPS)
A whole bunch of changes to the code (can see the git history for that)
Using elli instead of cowboy (much worse, ~2200 QPS).

What I’m hoping is that other people on this forum can grab the code and take a look for obvious problems, as well as potentially running “wrk” against it in whatever environments they have. I spent some time with eflame and Observer trying to figure out why things are slow but I’m not familiar enough with BEAM/Elixir to make any real headway into figuring out why things are so slow.

Any help would be really appreciated!

28 comments

#performance

2 10628 28

2016-06-23 21:54:30 UTC

Showing Posts 28 to 19

Show Best Posts
Show All (oldest first)
Show All (newest first)

chu

I was also looking at this and came across pastelli and pastelli_phoenix. Might be what you are looking for.

Post #28

Matt-Hornsby

@sasajuric Thank you so much for taking the time to put that blog post together! My tests are mostly pretty similar, but wasn’t able to get much above 10k RPS with 100 concurrent connections. I’m going to try to repro what you put together and see if I can replicate your performance numbers.[quote=“apenney, post:27, topic:473”]
So for anyone still trying this, I would definitely evaluate Elli over Cowboy in terms of raw performance.
[/quote]

Thank you for the update @apenney. Is it pretty easy to configure Phoenix to use Elli?

Post #27

apenney

I have a weird, semi useless update, as I haven’t really tried again due to a lack of time.

I did some experimentation with just raw cowboy (no Elixir) and had terrible results no matter what I did. I tried cowboy 2 as well, same bad results.

However, I did some experiments with raw Elli and somehow (I still don’t understand how, given +A 0 given to me by rebar3) I was able to get 160,000 QPS out of it running wrk on the same machine. I tried a Elli Plug adapter I found and got 2300 QPS so I didn’t continue down that path.

I also did some experiments with raw nginx to ensure I was doing something possible and around 160k was the limit I was able to get between the two servers I have. (Given the huge volume of POSTs that’s pretty great).

So for anyone still trying this, I would definitely evaluate Elli over Cowboy in terms of raw performance.

Post #26

sasajuric

Author of Elixir In Action

Inspired by this thread and the similar thread, I blogged about getting some low latency numbers with wrk. You can find the article here.

Post #25

jmilet

Maybe I’m wrong and this is not the problem you have, but we got the same performance problem and fixed it increasing the number of acceptors in the cowboy configuration. You can try it anyway.

Post #24

Matt-Hornsby

Here’s an update from @apenney on the thread I opened over on elixir-lang-talk:

Just a quick note from someone that tried (and failed) to get acceptable performance numbers from Elixir.

You should check into wrk2 instead of wrk1 for your testing. The reason being that wrk1 has a “coordinated omission” problem whereby the tool and the service accidently conspire to produce nicer numbers!

Wrk2 will send traffic at a steady rate instead of “when it can”, which often results in drastically different numbers. I was seeing latency in the seconds at the 99th percentile with wrk2 and a quick test with just plug.

When we tested our own scala based service internally we went from seeing 100ms at 100% to 9 minutes with wrk2, so it was quite an eye opener!

Sad that there wasn’t a happy ending

Post #23

bratsche

@apenney Is there any update on this? I’m curious if you were able to solve some of your performance problems.

Post #22

josevalim

Creator of Elixir

If it helps, here is my .kerlc file:

$ cat ~/.kerlrc
KERL_CONFIGURE_OPTIONS="--disable-debug --disable-silent-rules --without-javac --enable-shared-zlib --enable-dynamic-ssl-lib --enable-hipe --enable-sctp --enable-smp-support --enable-threads --enable-kernel-poll --enable-wx --enable-darwin-64bit"
KERL_DEFAULT_INSTALL_DIR=/Users/jose/.kerl/installs

If you run erl and paste here the beginning of the output, we will have an idea of what is enabled and not in your machine. IIRC, compiling Erlang without those flags is at least twice slower in my machine.

Post #21

apenney

The last part of your post caught my eye, @josevalim , (I haven’t managed to resolve the issue yet, the closest I’ve gotten is 160k using elli with raw erlang. I plan to try and tackle this again on the weekend).

I installed erlang both from “kerl” (on fedora23) and from the erlang-solutions package repo on Ubuntu. Are there additional optimizations of erlang that people don’t get out of the box that can be compiled in? I didn’t set any flags during the installation with kerl, just whatever the defaults were.

Post #20

josevalim

Creator of Elixir

This is definitely suspicious. Phoenix in dev is going to do code reloading and other things. If you are getting the same performance in both, there is definitely something else at play.

I would expect the default Phoenix stack in production to be around a 10% impact on a pure Plug stack. Also remember that logging has a huge impact in performance, make sure both apps are logging about the same amount of information.

If results are still suspicious, please let us know how you installed Erlang and which flags were used during the installation.

Post #19

Load more posts (18 remaining)