How is it possible to get 2 million websocket connections when you have 65536 available ports?

Questions / Help
1

I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% (65536). What exactly does that mean? How is it possible to go over the 65536 connections when that is the limit of ports?

image

1 Like
2

A quote from stackoverflow:

You misunderstand port numbers: a server listens only on one port and can have large numbers of open sockets from clients connecting to that one port.

On the TCP level, the tuple (source ip, source port, destination ip, destination port) must be unique for each simultaneous connection.
That means:

  • a single client cannot open more than 65535 simultaneous connections to a single server.
  • But, a server can (theoretically) serve 65535 simultaneous connections per client.

So in practice the server is only limited by how much CPU power, memory etc.

There’s a real example in The Road to 2 Million Websocket Connections in Phoenix - Phoenix Blog, they were using:

  • multiple general performance servers to play the role of clients:
    • if one server has only one fixed IP, and it only can provide 65535 source ports. Then, one server is able to open about 60K concurrent connections.
    • if you want to emulate 2million concurrent websocket connections, you need about 34 servers - 2_000_000 / 60_000 = 33.333333 ≈ 34
    • as a sidenote, if you have a powerful server, and have 34 IP assigned to this server, then in theory, you can simulate 2million concurrent websocket connections on this server, too.
  • a powerful server(40 Core CPU / 128GB RAM) to handle concurrent websocket connections:
    • it is not limited by the amount of its own available ports, because the source IP and source port is different for every connection.
8 Likes
3

Each network connection is described by tuple of 4 values:

  • Source Address
  • Source Port
  • Target Address
  • Target Port

So only if these 4 values match, we cannot open new connection. This should make it clear how we can have 2M (or more) simultaneous connections to single service listening on single port.

7 Likes
4

Coudflare goes into a good post on the topic https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/

4 Likes
5

Thanks for the great responses.