ostap
Investigating 502s from load testing my Phoenix app on Dokku
Finally deployed my Phoenix with SQLite3 app on Dokku. First things first, I did a lil wrk test from my laptop.
Wrk command with results and server info
The server is “RS 2000 G12” from Netcups. The endpoint serves an inertia-phoenix-powered controller-scaffolded login page. Default Dokku on Debian setup. No Nginx or kernel tuning applied.
$ wrk -t12 -c400 -d30s https://example.com/users/log-in
Running 30s test @ https://example.com/users/log-in
12 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 127.67ms 140.81ms 1.15s 89.49%
Req/Sec 98.90 81.46 535.00 82.60%
33595 requests in 30.07s, 64.23MB read
Non-2xx or 3xx responses: 5365
Requests/sec: 1117.32
Transfer/sec: 2.14MB
$ wrk -t12 -c400 -d30s https://example.com/users/log-in
Running 30s test @ https://example.com/users/log-in
12 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 202.20ms 287.43ms 1.96s 89.15%
Req/Sec 96.50 92.09 640.00 84.52%
32998 requests in 30.10s, 63.95MB read
Non-2xx or 3xx responses: 4778
Requests/sec: 1096.33
Transfer/sec: 2.12MB
$ wrk -t12 -c400 -d30s https://example.com/users/log-in
Running 30s test @ https://example.com/users/log-in
12 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 168.45ms 200.40ms 1.57s 86.71%
Req/Sec 100.09 86.62 650.00 83.00%
34932 requests in 30.10s, 64.79MB read
Non-2xx or 3xx responses: 6704
Requests/sec: 1160.69
Transfer/sec: 2.15MB
During the test, the server’s CPU usage went up to 50-70%, and the process using it the most was either the Beam container or Nginx. In Nginx, I have an SSL certificate to encrypt traffic from Cloudflare to the server.
After some time of running wrk, the server started responding with a 502 error. And I’m really curious to see why that is. I thought it would keep fighting until the CPU reaches 100%.
Even after the wrk is finished, it takes about 5-15 seconds for the server to change from 502s to 200.
Checking Dokku and Docker logs shows lots of similar request logs, which makes it hard to spot any errors:
09:52:34.997 request_id=GIUJwWJwbI8MW6cAAwso [info] GET /users/log-in
Anything I could do to find the reasons behind those 502? My first guess is adding Sentry or alike to see if there are any error logs besides the request logs. Will post an update.
Marked As Solved
ostap
I applied keepalive 64 to my Nginx proxy config and re-run wrk test. The throughput doubled, there were no 502 errors anymore 
upstream subkit-5000 {
server 172.17.0.3:5000;
keepalive 64;
}
It seems that without keepalive, there would be a ton of constant new TCP connections opened between Nginx and the Docker container, making it hit the machine’s Linux configuration limit:
P.S I’m yet to find out if my Nginx config gets overridden on every Dokku deploy, in that case I’ll consult Dokku docs on what’s the best way to apply my own config template.
Also Liked
rhcarvalho
(Load) testing can be fun and uncover a lot of interesting lessons!
One tip is reading what Gil Tene wrote on the topic and several talks on YouTube (in particular about coordinated omission).
At some point you need to consider also what your goals are with the load testing, and when to stop.
Do you expect to serve 400 concurrent users with your current setup?
Hitting your controllers is only one part of the test. Consider also WebSocket persistent connections (there will be client-Cloudflare, Cloudflare-nginx, and nginx-BEAM connections open). Consider database writes according to your expectations.
Have fun and learn a bunch!
rhcarvalho
Are you hitting Cloudflare in your tests or bypassing it and hitting your server directly?
502 is typically Error 502 or 504 · Cloudflare Support docs
You also need to get familiar with your nginx configuration Nginx Proxy - Dokku Documentation
You need to consider all hops in your setup before reaching the BEAM.
Sentry will not help you in this particular case (i.e. networking and errors before your application sees a request).
rhcarvalho
Yep, look at Nginx Proxy - Dokku Documentation
The good thing is you can keep your nginx config along with your app’s source code.
Last Post!
ostap
I contributed proxy-keepalive option. Now we can easily set it without using a custom config file:
dokku nginx:set --global proxy-keepalive 64
Popular in Questions
Other popular topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Phoenix, LiveView, multimedia pipelines, distributed systems. 400 engineers, Elixir since 2017. Available for hire.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #elixirconf-us
- #advent-of-code
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #hex
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking for Elixir devs who love to ship. Start your free account.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"