jaybe78

jaybe78

Ditributed cluster: up to how many nodes we can scale using a library like libcluster?

Hello,

I would like to know up to how many nodes max we can run a distributed app with a simple libray like libcluster ?

I read somewhere that after 40/50 nodes, you can start having issues like loss of messages and it generally does not scale well above that number.

For the context:

I’m currently building an app with an expectation of 2/3 millions concurrent users.
Those users would spread around 300 000 + channels.
So each channel would contain something like 100 users or less joining and they would mostly exchange chat messages.

Phoenix Presence is set up on those channels so an user can be aware at anytime of whose connected.

I heard about an Erlang library called Partisan that is used for very large cluster, but I guess for my use cases lib cluster should be enough ?

If anyone has been working in a distributed app it would be nice to share your feedback here.

Cheers.

Most Liked

mpope

mpope

Fair enough, thought for food.

For automated discovery specifically take a look at GitHub - nerves-networking/mdns_lite: A simple, no frills mDNS implementation in Elixir · GitHub

jaybe78

jaybe78

yes thanks definitely interesting :slight_smile:

pleb

pleb

Virtual nodes are spread across physical nodes, and the dynamic partitioning scheme allows physical nodes to be added and removed while virtual nodes always find the closest physical server in the event that a node is removed to recover, or spread out when a new node is added.

Although this scheme might not work too well, if the number of participants is not uniform, and one partition is overwhelmed, as was the case with How Discord Scaled Elixir to 5,000,000 Concurrent Users implementing a similar scheme

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
marius95
Hello everyone, I try to use an Javascript Event Handler in my root.html.leex file. Therefore I created a function in the app.js file: ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
earth10
Hi, I’m just starting to build a side-project with Elixir and Phoenix and doing some basic test with Elixir alone. What strikes me is th...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New
jononomo
For some reason my phoenix channels are working for me in my local dev environment, but as soon as I deploy via Docker, I get a 403 error...
New

Other popular topics Top

danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 29603 241
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
Emily
I have VueJS GUIs with the project generated using Webpack. I have Elixir modules that will need to be used by the VueJS GUIs. I forese...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3271 127089 1222
New
hariharasudhan94
Lets say I have map like this fetching from my database %{"_id" => #BSON.ObjectId<58eb1a7a9ad169198c3dXXXX>, "email" => ...
New
jaysoifer
Is there a way to rollback a specific migration and only that one (“skipping” all the other ones)? Would mix ecto.rollback -v 200809061...
New
vegabook
I’m brand new to Phoenix and I have stripped one of the demo applications to the bone. I just want to get an svg up on the screen. Here i...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement