invacto
Restarting an application using websockets
Hey,
When restarting or upgrading and application using web sockets.
How do you deal with the fact that after restart all web sockets will simultaneously attempt to reconnect and overload your system.
Marked As Solved
benwilson512
The traditional answer here is to use rolling deploys over a set of M servers, and only have N of those changed out at any given moment. You pick N such that N/M doesn’t overload the system. As a concrete example if you’re running 10 servers, you might do a rolling deploy where you are shifting 2 of those at any given moment. 2/10 servers worth of traffic reconnecting would need to be what your cluster could handle.
The lower your overall traffic, the lower those numbers can be, it just depends on your load profile.
There are fancier answers as well these days where you instead of hard starting / stopping any particular server, you spawn an entire new set of 10, and then your load balancer gradually shifts traffic from the old set to the new set.
NOTE: I’m using the term “servers” here sort of loosely, I really mean “running instances of your application”, in the context of containers these may or may not map to individual whole computers somewhere.
Also Liked
sb8244
There’s another aspect of this that I feel strongly about because it’s brought my production down before.
If you use tokens to connect to the WebSocket and the tokens are fetched from a service using async request (either the WebSocket service itself or another in the cluster), then you must be prepared for your real-time application crashing and causing every user to try to refresh their token. This can cause a massive spike in requests in a short period.
I must use a short lived token in my app, so I solved this by fetching a token every 5-9 minutes. I trade off fetching a token I may not need for stability of the token service.
I never had an issue in rolling deployments because the number of users affected at once was smaller, but I did experience issues when the service unexpectedly started having operation issues.
Real-Time Phoenix’s chapter on deployment should be going into beta in the next 2 weeks and includes a bit on this topic.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Deploy Phoenix to your own VPS with PaaS ease: push-to-deploy, zero downtime, auto SSL, managed Postgres. 14-day free trial at potions.io
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #javascript
- #code-sync
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Real-time error tracking, performance insights, and observability for devs.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"