markmeeus
Phoenix Channels horizontal scalability
Hi Everyone,
I have a question regarding horizontal scalability of the phoenix channels.
We are currently using phoenix channels to deliver realtime information to our app’s frontend.
A large portion of this data is a continuous flow of updates from an external system. Something like GPS updates.
Right now, everything is running real smooth with a few hundred concurrent users on a 2 small servers (using PG2 adapter).
We are preparing the software to deliver 100 times the data to a few 10 thousand users.
However, a specific subset of users will only need a specific subset of the data.
If I understand correctly, if we scale out to 10 servers, all servers will need to receive and process all that data, because the users’ sockets are spread out over all instances of the phoenix app.
So my question is, isn’t there a point where horizontal scalability ends because the data load is too high, and would it be a good idea to create separate clusters of the phoenix app, serving the sub-set of data to the subset of users? Or is there an easier solution I’m missing?
Trending in Questions
Other Trending Topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #elixirconf-us
- #supervisor
- #advent-of-code
- #distillery
- #processes
- #api
- #forms
- #metaprogramming
- #performance
- #security
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Error tracking, logs, uptime, and Just Enough APM. Built for developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
The team behind Membrane, Popcorn, LiveDebugger. Available for hire.
Courses that'll move you from confusion to "Aha, now I get it!"
First 4 of 4 Posts!
idi527
Probably. You can find this point with some load testing. Tsung supports websockets, so you can model your users’ interactions with it.
benwilson512
Notably, Phoenix also scales vertically extremely well. You’ve probably seen this: The Road to 2 Million Websocket Connections in Phoenix - Phoenix Blog
The core concern is sensible, in that the erlang distribution scheme does have limits to how far out horizontally you can scale a cluster (10 however should be fine). However, the load you should be able to handle on a reasonable number of servers should probably last you for a long time.
cerdiogenes
I just made a playground for this, take a look: GitHub - kdiogenes/phoenix_channels_horizontal_scalability: Simple scaffold to play with phoenix channels in an auto-scaling environment · GitHub
idi527
A proper way to load test though would be not docker on a laptop, but a couple of vms in the cloud … Just in case it comes in helpful, I have a demo libcluster+digitalocean+terraform project. It can be modified to support phoenix, and then something like tsung can be used to actually load test it.
Also GitHub - dsander/phoenix-connection-benchmark: Reproduction of the 2 million connection in Phoenix benchmark · GitHub might still come in handy.