Evaluating Elixir / Phoenix for a web-scale, performance-critical application

I’ve read several times that for performance testing you should run the app in production mode.

MIX_ENV=prod mix compile.protocols
MIX_ENV=prod PORT=4001 elixir -pa _build/prod/consolidated -S mix phoenix.server

This is from the 0.7.2 docs so I’m not sure it’s all still needed, but might be worth a try.

Thanks for the great suggestion @terakilobyte - I just tried running it in PROD mode:

MIX_ENV=prod mix compile
MIX_ENV=prod mix phoenix.digest
MIX_ENV=prod PORT=4001 mix phoenix.server

./wrk -t8 -c100 -d30S --timeout 2000 http://localhost:4001/api/products

Running 30s test @ http://localhost:4001/api/products
8 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 29.16ms 32.83ms 365.59ms 94.76%
Req/Sec 494.23 139.45 770.00 76.14%
116284 requests in 30.06s, 71.44MB read
Requests/sec: 3868.18
Transfer/sec: 2.38MB

Still not very good unfortunately

Interestingly, with only 10 concurrent connections:

Running 10s test @ http://localhost:4001/api/products
8 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.18ms 3.91ms 40.59ms 88.15%
Req/Sec 1.12k 249.24 1.78k 68.38%
89005 requests in 10.01s, 54.68MB read
Requests/sec: 8890.15
Transfer/sec: 5.46M

There was a similar thread recently. You may find some tips there.

Some low-hanging fruit to improve perf would be:

1. Raise log level in prod to :warn to suppress logging each request.
2. If you’re testing a REST endpoint, make sure it goes through the :api pipeline, and not the :browser one.
3. Build an OTP release and bench against that

Also, check your cpu usage while testing. If all your CPUs are not constantly near 100% then you may have some bottleneck.

Check this micro benchmark tool released by a member of the community: Benche and BencheeCSV 0.1.0 release - easy and extensible (micro) benchmarking

To expand on Sasa’s point, the :browser endpoint by default generates CSRF tokens to be used in forms as a security measure. These can be fairly expensive to generate at least compared to a hello_world JSON endpoint. Definitely make sure you aren’t doing that.

Thank you for your quick response on this @sasajuric. I looked around in the forums first but didn’t see the other thread. Lots of great information over there to try. Pretty sure I’m piping through :api - here’s my router:

defmodule MarketApi.Router do
  use MarketApi.Web, :router

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/api", MarketApi do
    pipe_through :api
    resources "/products", ProductController, except: [:new, :edit]
  end
end

CPU is pretty well maxed out across all cores when I run the tests.

Thanks @benwilson512 - I’m super green to Phoenix, so I appreciate the tip. I think I am using :api. Is there anything I need to do besides pipe_through :api ?[quote=“thinkpadder1, post:6, topic:832, full:true”]
Check this micro benchmark tool released by a member of the community: Benche and BencheeCSV 0.1.0 release - easy and extensible (micro) benchmarking
[/quote]

I hadn’t seen this before - I’ll give it a whirl. Is it possible with the tool to have it benchmark where Phoenix is spending all its time?

Try to make a release with exrm, and then start as a service:
rel/your_app/bin/your_app start.

Also, check out Elixir’s very own Observer tool.

Could you put your test app to Github or smth else? It would help to found a reason.