Also I just want to reiterate for the third time that I think Redis is a perfectly valid solution and I take no issue, at all, with you recommending it. If you read my reply to the OP carefully you will note I also included it amongst my recommendations before attempting to build a distributed system 
What I took issue with was this:
This is false. And I was not even the first person to point this out, it was @LostKobrakai - I was mostly just clarifying the point he made, since his reply was quite terse.
1 Like
Have you consider prototyping this in mnesia, it would give you everything out of the box for something this simple.
Obviously you could build your own leveraging ets, global and etc. but feels like mnesia would be a better fit for this. Just my 50 cent after a cursory read.
There are also atomics — erts v16.0 and counters — erts v16.0
Anyways, my recommendation is always to look into OTP with these kind of questions, since they likely have been solved before! Best of luck!
From the words of the maintainer of Cachex
The intent for Cachex has always been caching of data; I do not intend to get into the world of supporting distributed databases. The idea of adding “distributed” caches in the first place was so a newly added Node B would be able to fetch pre-warmed data from Node A. If a user wants persistent distributed cache, the answer exists with (e.g.) Redis and Cachex being a local hot layer over a single remote
1 Like
What are the rooms for? What do they contain? Messages? State? Where is that information stored?
I’m holding a count for each channel with themselves contain a list of user_id.
Do you need fault tolerance, or are the rooms ephemeral?
Yes and no.
The room lives as long as people joins or stays in the room.
Those rooms for users to get people in touch with another person, so once someone found a partner, they live the room.
So on average people would join and leave regularly, but you could have more people just in the meantime
Do you need to scale out?
Yes
If not, you can just put all of the rooms in one ETS table on one node. Or in Postgres, or Redis, or whatever. That will probably scale into the millions.
If you look at this post
This is kind of what I’ve done.
I merge the local counters diff with Phoenix trackers and then store the result in an ETS table.
If you do need to scale out, hash partition the rooms across nodes. When a user connects, choose a random node and ask that node to add the user to one of its rooms. The node knows the state of each room on that node, so it can pick one at random. Then if you need to look up any other state about that room, you can hash it and contact its node.
I create room ids on demand in ascending order based on the last room filled.
if all rooms from 0 to 10 are full, I assign the next users to topic “room#11”
That allows me to spread users evenly among new topics.
Not sure to understand how assigning users to random topic in random node, would work ?
Wouldn’t I end up with lonely users in many different topics ?
So before I suggest anything else, I will say one last time that you should at least try a fully centralized approach first, where you store all topic metadata in one ets table on one node (or in Redis, if you want). If you have 1 million users with 100 users per room that would only be 10,000 keys, which is nothing. If you have one or two orders of magnitude more than that maybe it becomes a problem. I will assume you know what you’re doing, though, and move on
What I am suggesting here is that you turn this consistency problem inside-out. Instead of trying to store an eventually consistent “user count” of each room on every node, you should store some room counts, un-replicated, on each node. So each room has a node that it “lives” on, which maintains its count. This is called partitioning, or sharding.
If the topics are external and out of your control, like say chat channels where the user inputs a topic (“programming”) and joins that room, then you need some sort of routing table, or you can use hash partitioning. That’s what I mentioned before.
However, if you have total control of the ids, which it sounds to me like you do, then you don’t even need to do that. You can simply add a node id to each room id, like room:node2:10, and then you know exactly where to send requests regarding that room.
You can store user counts, ids, and whatever other metadata in an ETS table on the node that room lives on. That side-steps most of the difficult consistency problems you are running in to.
So before I suggest anything else, I will say one last time that you should at least try a fully centralized approach first, where you store all topic metadata in one ets table
Yes I totally agree and this is what I’ve done done so far.
I basically store all the counts in 1 ETS table and I have a balancer algorithm running with highlander (only executed in a single node across the whole cluster doing the balancer for all rooms)
What I am suggesting here is that you turn this consistency problem inside-out. Instead of trying to store an eventually consistent “user count” of each room on every node, you should store some room counts, un-replicated, on each node. So each room has a node that it “lives” on, which maintains its count. This is called partitioning, or sharding.
Sharding sounds quite nice yes and I thought about it… BUT the only issue is that if I don’t store all the room counts in the same ETS but partitionned them across nodes, that means my balancer which re-equilibrates rooms won’t be able to do it “globally”, but only with the rooms in its nodes, so there might some room left out…
For example:
Node 1, you could have :
room-1 => 30 users
room-2 => 40 users
Node 2, you could have :
room-3 => 15 users
room-5 => 20 users
Node 3, you could have :
room-4 => 5 users left
Since rooms are partitioned, my balancer won’t be able to merge room-4 with rooms 1, 3 and 5.
Instead
balancer one node 1 will merge 1 and 2 together
balancer one node 2 will merge 3 and 5 together
Room 4 cannot be merged with anything.
There are far more rooms than nodes, so I don’t think it would be too big a deal. But if you have enough nodes that you start to get some skew from the distribution, you can use the same load balancing algorithm across nodes. Pick two nodes and route the join to the emptier one.
You can keep an eventually consistent “load” for each other node on each node. There are only a handful of nodes so this is much easier to keep in sync by gossiping, and having exact information matters much less because the load on a particular server is not going to change by that much in a few seconds.
A simple way would be to destroy room 4 and send all of its users through the routing algorithm separately. But if you want it to work as you say, there is no reason you can’t add users to rooms “as a batch”. It’s just more logic in the load balancer.
The logic here is not trivial though. For example, do you want to merge room 4 off of that node? Or do you want to take some users from the other rooms and send them to room 4? After all, if you remove those users from node 3, it will be empty, which is not very balanced!
In practice, since there are so many more rooms than nodes, I don’t think you would run into this problem. There should be enough rooms on each node that you can balance them entirely intra-node. If not, get rid of some nodes!
