Implementing distributed user counters

There are far more rooms than nodes, so I don’t think it would be too big a deal. But if you have enough nodes that you start to get some skew from the distribution, you can use the same load balancing algorithm across nodes. Pick two nodes and route the join to the emptier one.

You can keep an eventually consistent “load” for each other node on each node. There are only a handful of nodes so this is much easier to keep in sync by gossiping, and having exact information matters much less because the load on a particular server is not going to change by that much in a few seconds.

A simple way would be to destroy room 4 and send all of its users through the routing algorithm separately. But if you want it to work as you say, there is no reason you can’t add users to rooms “as a batch”. It’s just more logic in the load balancer.

The logic here is not trivial though. For example, do you want to merge room 4 off of that node? Or do you want to take some users from the other rooms and send them to room 4? After all, if you remove those users from node 3, it will be empty, which is not very balanced!

In practice, since there are so many more rooms than nodes, I don’t think you would run into this problem. There should be enough rooms on each node that you can balance them entirely intra-node. If not, get rid of some nodes!

So before I suggest anything else, I will say one last time that you should at least try a fully centralized approach first, where you store all topic metadata in one ets table

Yes I totally agree and this is what I’ve done done so far.
I basically store all the counts in 1 ETS table and I have a balancer algorithm running with highlander (only executed in a single node across the whole cluster doing the balancer for all rooms)

What I am suggesting here is that you turn this consistency problem inside-out. Instead of trying to store an eventually consistent “user count” of each room on every node, you should store some room counts, un-replicated, on each node. So each room has a node that it “lives” on, which maintains its count. This is called partitioning, or sharding.

Sharding sounds quite nice yes and I thought about it… BUT the only issue is that if I don’t store all the room counts in the same ETS but partitionned them across nodes, that means my balancer which re-equilibrates rooms won’t be able to do it “globally”, but only with the rooms in its nodes, so there might some room left out…

For example:

Node 1, you could have :
room-1 => 30 users
room-2 => 40 users

Node 2, you could have :
room-3 => 15 users
room-5 => 20 users

Node 3, you could have :
room-4 => 5 users left

Since rooms are partitioned, my balancer won’t be able to merge room-4 with rooms 1, 3 and 5.

Instead
balancer one node 1 will merge 1 and 2 together
balancer one node 2 will merge 3 and 5 together
Room 4 cannot be merged with anything.

So before I suggest anything else, I will say one last time that you should at least try a fully centralized approach first, where you store all topic metadata in one ets table on one node (or in Redis, if you want). If you have 1 million users with 100 users per room that would only be 10,000 keys, which is nothing. If you have one or two orders of magnitude more than that maybe it becomes a problem. I will assume you know what you’re doing, though, and move on

What I am suggesting here is that you turn this consistency problem inside-out. Instead of trying to store an eventually consistent “user count” of each room on every node, you should store some room counts, un-replicated, on each node. So each room has a node that it “lives” on, which maintains its count. This is called partitioning, or sharding.

If the topics are external and out of your control, like say chat channels where the user inputs a topic (“programming”) and joins that room, then you need some sort of routing table, or you can use hash partitioning. That’s what I mentioned before.

However, if you have total control of the ids, which it sounds to me like you do, then you don’t even need to do that. You can simply add a node id to each room id, like room:node2:10, and then you know exactly where to send requests regarding that room.

You can store user counts, ids, and whatever other metadata in an ETS table on the node that room lives on. That side-steps most of the difficult consistency problems you are running in to.

What are the rooms for? What do they contain? Messages? State? Where is that information stored?

I’m holding a count for each channel with themselves contain a list of user_id.

Do you need fault tolerance, or are the rooms ephemeral?
Yes and no.
The room lives as long as people joins or stays in the room.
Those rooms for users to get people in touch with another person, so once someone found a partner, they live the room.
So on average people would join and leave regularly, but you could have more people just in the meantime

Do you need to scale out?
Yes

If not, you can just put all of the rooms in one ETS table on one node. Or in Postgres, or Redis, or whatever. That will probably scale into the millions.

If you look at this post
This is kind of what I’ve done.

I merge the local counters diff with Phoenix trackers and then store the result in an ETS table.

If you do need to scale out, hash partition the rooms across nodes. When a user connects, choose a random node and ask that node to add the user to one of its rooms. The node knows the state of each room on that node, so it can pick one at random. Then if you need to look up any other state about that room, you can hash it and contact its node.

I create room ids on demand in ascending order based on the last room filled.
if all rooms from 0 to 10 are full, I assign the next users to topic “room#11”

That allows me to spread users evenly among new topics.

Not sure to understand how assigning users to random topic in random node, would work ?
Wouldn’t I end up with lonely users in many different topics ?

From the words of the maintainer of Cachex

The intent for Cachex has always been caching of data; I do not intend to get into the world of supporting distributed databases. The idea of adding “distributed” caches in the first place was so a newly added Node B would be able to fetch pre-warmed data from Node A. If a user wants persistent distributed cache, the answer exists with (e.g.) Redis and Cachex being a local hot layer over a single remote

Have you consider prototyping this in mnesia, it would give you everything out of the box for something this simple.

Obviously you could build your own leveraging ets, global and etc. but feels like mnesia would be a better fit for this. Just my 50 cent after a cursory read.

There are also atomics — OTP 29.0.2 (erts 17.0.2) and counters — OTP 29.0.2 (erts 17.0.2)

Anyways, my recommendation is always to look into OTP with these kind of questions, since they likely have been solved before! Best of luck!

Also I just want to reiterate for the third time that I think Redis is a perfectly valid solution and I take no issue, at all, with you recommending it. If you read my reply to the OP carefully you will note I also included it amongst my recommendations before attempting to build a distributed system

What I took issue with was this:

This is false. And I was not even the first person to point this out, it was @LostKobrakai - I was mostly just clarifying the point he made, since his reply was quite terse.

I was simply trying to answer your question in earnest. Order matching is an example of a similar situation (orders join and leave the book) where strong consistency is required and the system effectively cannot be partitioned. If I had posted that question as you had I would have been interested in that response, so I gave it. If you were not interested, that’s perfectly fine; you could have simply ignored it!

Personal attacks aside (which I do not appreciate, regardless of who they’re aimed at), you are literally confusing me with someone else. That was not me.

You say no one is holding the absolute truth, but thats my critisism of the way he always responds to people.

The other day he was arguing with me because he has his own personal “correct” opinion of what the term “infinitescroll” means, that is different from the commonly understood term which resulted in a pointless discussion. He constantly argues with things you have never said, takes what you have said and either adds meaning to it, or tries to find meaning you never tried to exress and generally just nitpicks random things that are not relevant to the topic being discussed, or helpful in solving it.

Everyone has the right to disagree, but I have the right to get annoyed when people respond to me like this. The goal of a thread like this is to solve the main issue in the main post, not nitpick every little nuanced thing someone says or try to 1up people constantly.

I mean, what is the point of this? It has no relevance to anything being discussed in the thread at all but gets directed at me. What if people in the future come along looking for help solving the posts issue, then end up wasting time researching the things in his response?

Hi, maybe distributed cachex is an option to check: