I’ve been trying load test of phoenix channels for an app I’ve been working on for a client. I’m using tsung to do so. I’m struggling to get more than 50K-60K connections at a time. I feel like I’m missing something simple, but I’ve been struggling with this for a while now.

EDIT: I also just tried it with a brand new phoenix project:
GitHub - praveenperera/bare_channel · GitHub

Here is my info:

Tsung setup:
Version: 1.6.0
5 worker nodes: DigitalOcean - 16GB 6vCPUs
1 controller node: DigitalOcean - 16GB 6vCPUs

Application server setup:
DigitalOcean - 8GB 4vCPUs also tried 16GB 6vCPUs

Elixir: 1.8.1
Phoenix: 1.4.6

I removed all code from the channel. Now on join it just sends back an :ok response with an empty payload.

What I’ve tried so far:

1. Increasing limits on all the workers, the controller and the node running the phoenix application. I used this script to do so:

#!/bin/bash

#limits
sudo sysctl -w fs.file-max=12000500;
sudo sysctl -w fs.nr_open=20000500;
ulimit -n 20000001;
sudo sysctl -w net.ipv4.tcp_mem='10000000 10000000 10000000';
sudo sysctl -w net.ipv4.tcp_rmem='1024 4096 16384';
sudo sysctl -w net.ipv4.tcp_wmem='1024 4096 16384';
sudo sysctl -w net.core.rmem_max=16384;
sudo sysctl -w net.core.wmem_max=16384;

echo "fs.file-max = 1048576" >> /etc/sysctl.conf
echo "# limits
* soft     nproc          1048576
* hard     nproc          1048576
* soft     nofile         1048576
* hard     nofile         1048576
root soft     nproc          1048576
root hard     nproc          1048576
root soft     nofile         1048576
root hard     nofile         1048576
" >> /etc/security/limits.conf
echo "session required pam_limits.so" >> /etc/pam.d/common-session
sysctl -p

touch /root/setup_log
echo "COMPLETE AT: $(date)" >> /root/setup_log

2. Different versions of tsung, different settings.
3. Tried the websocket instead of tcp on tsung but it wasn’t sending messages properly
4. Switched over to using releases instead of doing mix phx.server
5. Removed all application code, only replying with an :ok and empty payload
6. Deploying a new version straight on the VPS, avoiding docker and k8s
7. Turned logger level to warn

<?xml version="1.0"?>
<tsung loglevel="warning" version="1.0">
  <clients>
    <client host="tsung-worker-1" use_controller_vm="false" maxusers="64000" cpu="6">
      <ip scan="true" value="eth0"/>
    </client>
    <client host="tsung-worker-2" use_controller_vm="false" maxusers="64000" cpu="6">
      <ip scan="true" value="eth0"/>
    </client>
    <client host="tsung-worker-3" use_controller_vm="false" maxusers="64000" cpu="6">
      <ip scan="true" value="eth0"/>
    </client>
    <client host="tsung-worker-4" use_controller_vm="false" maxusers="64000" cpu="6">
      <ip scan="true" value="eth0"/>
    </client>
    <client host="tsung-worker-5" use_controller_vm="false" maxusers="64000" cpu="6">
      <ip scan="true" value="eth0"/>
    </client>
  </clients>
  <servers>
    <server host="134.209.166.67" port="9962" type="tcp"/>
  </servers>
  <load>
    <arrivalphase phase="1" duration="300" unit="second">
      <users maxnumber="2000000" arrivalrate="2000" unit="second"/>
    </arrivalphase>
  </load>
  <options>
    <option name="ports_range" min="1025" max="65535"/>
  </options>
  <sessions>
    <session name="websocket" probability="100" type="ts_websocket">
      <request>
        <websocket type="connect" path="/socket/v1/websocket?vsn=2.0.0"/>
      </request>
      <request>
        <websocket type="message">          ["1", "1", "payload:url!!!commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4","phx_join", {}]        </websocket>
      </request>
      <for var="i" from="1" to="1000" incr="1">
        <thinktime value="30"/>
      </for>
    </session>
  </sessions>
</tsung>

Results:

29%20PM2666×896 97.4 KB

23%20PM1152×824 61.6 KB

I am only hitting about 60% CPU usage and 50% memory usage. Does anyone know how many connections I should be reasonably be expecting on a server this size?