How to limit the amount of active Websockets per node on Phoenix

Phoenix Forum Questions / Help
, , ,
1

On a LiveView app with 25+ million active users per month.

We are using K8s so I am going to use the word “pod”.

I have been told that when we experience an outage, we have to start all pods at the same time because if not, the first pod that becomes healthy will accept all the traffic from all the LiveViews trying to re-connect and run out of RAM and crash. Even if we gave that pod enough resources to not run out of RAM, most requests would time out waiting for a DB connection because our data stores can’t handle all of our LiveView sockets re-connecting at the same time.

I thought it would be reasonable to set a limit for how many connections a Phoenix server should accept.

We are using Cowboy and I believe this should be handled by the HTTP server.

I found the Plug.Cowboy docs

Which led me to try this in the Endpoint:

      http: [
        port: 4000,
        transport_options: [max_connections: 1],
        ...
      ],

I was able to still connect several LiveViews locally. Then I looked at the ranch docs which shows that max connections is a “Soft limit”. Which means that I won’t be able to test if I configured it correctly or assure anyone of how it will behave in production.

Is there another approach that I should be considering to solve this problem? Would switching to Bandit make it simpler to achieve?

2

Hello,

I don’t know but I bet there are options in k8s to do that kind of configurations. So maybe you can put a hard limit here.

Or otherwise if you need to wait for other pods you could use the readiness probe to return “not ready” as long as there is not at least N nodes in the cluster. I believe that if the liveness probe is OK then k8s will start the next pod (when starting multiple pods / upscaling) even if the readiness probe is not OK.

It looks like this should be handled at the infrastructure level but I obvioulsy don’t know your system.

Anyway, I guess the “soft limit” should not be too far from the set limit. Maybe that soft limit may not be a problem in your specific case. But maybe it will not be able to counter a huge peak.

I suggest to read the code and figure out :slight_smile:

I was able to still connect several LiveViews locally

max_connections is the number of connections by listener. What is your num_acceptors? Also maybe you could try to open 100 websockets at the same time to see if you can open twice as much or just 103 or something.

3

This is my first thought, as well. k8s gives you plenty of tools to accomplish this, including aforementioned startup/readiness/liveness probes.

So why not start all pods at the same time? Maybe it’s a StatefulSet that starts up sequentially? But then you could still use probes to wait for several pods to be ready. But then we still have to worry about…

…which sounds like maybe a good opportunity for exponential backoff + jitter on the clients.

All fun problems :smiley:

4

I found a cowboy issue saying that max connections hold the HTTP connections waiting for upgrade to websocket, but after the upgrade they remove it from the max_connections count, which explains the soft limit.

Not out of the box I think, but it isn’t hard to fork the code and close the connection whenever you are at your pod limit.

But I agree with the others in this thread that this should be handled at ingress and not at the application level, a circuit breaker should be a good start, traefik for example, has a middleware for circuit break that can stop the requests when a percentage are above a certain latency.

5

My first thought agrees with all the responses that suggest doing this at the infrastructure layer. I have been bitten by the hard connection limit on Fly.io being set too low by default and limiting the number of concurrent LiveView users, so I know that infra can successfully do this. The cloud architect where I work suggested that we add Nginx in the Docker container and limit it there because this is how it has been successfully handled in other languages, let’s keep things standard and uniform, right? I think we can all agree that would work, so I can’t push back and say that won’t work, I can only say that “We don’t do that in the Beam community because Nginx can’t do anything for the Beam that the Beam can’t do for itself”. In summary, if I am not willing to do Elixir in the way that our cloud team wants me to do it (with an Nginx sidecar) then it becomes hard for me to tell them how to do infrastructure.

Someone at work read this mentioned to me that even better than a hard limit would be a queue. Each node might happily handle X concurrent LiveView clients, and so setting a hard limit at X or lower could cause us to drop requests that could have successfully been handled, while our data stores can’t hanlde X connections at the same time so setting it at X or higher will provide a bad user experience at times.

I’ll probably start a new question to ask how to put LiveView socket connects in a queue so that it can only succeed at a certain rate.

The hard limit would still be useful to prevent OOM. A fork of Bandit feels doable, but it would be better if there was a solved approach in the ecosystem for limiting max LiveView sockets. A Bandit config option would be preferred.

6

I think that’s a quite shortsighted idea. Yes the BEAM doesn’t need nginx in front. That doesn’t however mean that nginx in front of the BEAM cannot be useful. If bandit / the beam doesn’t come with a feature that doesn’t need much work on nginx, why not put nginx in front.

Though I don’t think the message here was to push this in a random sidecar, but more specifically the load balancer, which already needs to deal with knowing where to send traffic and proxing the connections.

And for the larger question of how to prevent overload – a random limit on number of connections is a quite naive way of dealing with that.

7

I just ran into @chrismccord at ElixirConf and posed this question to him.

He mentioned that we can define a connect callback to override the default in Phoenix.LiveView.Socket and even showed me the test

In that callback we can check if we are past the max limit to prevent OOM and we can also block and wait for a queue so that we only allow a configured amount of requests per second to the data stores.

3 Likes
8

I mean even without a callback, you could rather easily define a plug that would keep the number of active connections.

9

Blockquote
I think that’s a quite shortsighted idea. Yes the BEAM doesn’t need nginx in front. That doesn’t however mean that nginx in front of the BEAM cannot be useful. If bandit / the beam doesn’t come with a feature that doesn’t need much work on nginx, why not put nginx in front.

Thanks for the feedback @LostKobrakai I agree after re-reading it, the way I worded that is too broad to be true. In this situation the Beam can shed this load for itself so I don’t want to add an entire installation and configuration of a technology to do this one thing that a few lines of code could be doing.

10

@D4no0 I like the idea of just creating a plug.

I guess the question to answer is:

“Do we want to treat LiveView socket connects different than other traffic?”

I have a dream forming of a “Health.serve_new_traffic?” function that can very quickly read a boolean out of ETS, and then the boolean is updated by something that monitors the DB connection queue and system RAM usage and Erlang run queue.

If the node is in a state where it can not handle new traffic, it should probably drop all traffic anyways, regardless if it is an initial LiveView render or a LiveView socket connect or a Controller route.

If we rate-limit how many sockets can connect per second, our use-case would benefit from only applying that to LiveView socket connections.

Then again, maybe the plug could look at the conn and see that it is a LiveView socket connection and figure that into the decision on handling or dropping the traffic.

1 Like