Load Testing a Phoenix app using siege

Are you running wrk and siege from your local machine when testing the app on gigalixir? That wouldn’t probably show any useful data, since the network between your local machine and the server is not under your control.

So you’d need to run wrk and siege from within gigalixir, preferably the same AZ as your app.

Maybe The Road to 2 Million Websocket Connections in Phoenix - Phoenix Blog would help.

In addition to @idi527’s great comments, it would be helpful if you supplied some information. How much CPU / RAM do your servers have, what parameters are you passing to Siege / Wrk?

siege -d3 -c100 https://my.app/
wrk -t12 -c400 -d30s https://my.app/

The error that comes back most frequently from siege is

[error] socket: unable to connect sock.c:249: Operation already in progress

Sample response from WRK:

  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    91.13ms   19.46ms 343.70ms   76.15%
    Req/Sec   360.04     49.02   595.00     76.08%
  128189 requests in 30.10s, 58.92MB read
Requests/sec:   4258.29
Transfer/sec:      1.96MB

Sample siege output:

Transactions:		         761 hits
Availability:		      100.00 %
Elapsed time:		       14.66 secs
Data transferred:	        0.01 MB
Response time:		        0.50 secs
Transaction rate:	       51.91 trans/sec
Throughput:		        0.00 MB/sec
Concurrency:		       25.74
Successful transactions:         761
Failed transactions:	           0
Longest transaction:	        2.57
Shortest transaction:	        0.15

The Gigalixir instance has 0.5 GB Memory and 1 Replica. I’m not sure what that has for CPU.

I’m not sure what this means but this sounds more like a siege issue? 4k req/ sec from wrk on what is probably 1 CPU is really rather good particularly given that you’re also doing it over the general web.

I’d recommend that you have a look at wrk2 (GitHub - giltene/wrk2: A constant throughput, correct latency recording variant of wrk · GitHub) as well. It is especially written to consider something called coordinated omission (explained in the readme for wrk2 and there are some good talks on it as well if you search for it).

It also allows you to set the rate (perhaps wrk can do this today as well) of requests so that you don’t overload your system (unless that is what you are testing).

For more elaborate testing you have

• tsung (http://tsung.erlang-projects.org/). It feels a bit “old” but it pretty good. I think this is what was used when they tested the 2M web socket connections.
• gatling (https://gatling.io/) I found quite pleasant to work with.

I recently discovered k6 – it’s pretty clean and seems to offer similar scripting / tests as Gatling.