Implementing distributed user counters

[jaybe78]

Hello,

I’ve recently developed a module that creates distributed user counters, based on what’s described in this post

Basically, locally I use ETS table to atomically know how many people there is in a channel, then I aggregate all counters of all nodes using Phoenix Tracker.

In summary, users starts by joining “room:lobby”, using ETS table, I check what’s the last room id with available slots( <= 70), and if I can’t find one, I assign them to a new topic.

Those topics (“room#{id}”) are created in ascending order (“room#1”, “room#2”, …), and I use the same topics on every nodes.

With that in place I’m able to create channel topics with a max size of 70 or so users.

It works quite well overall, but I have a “problem” which concern the total number of users on channel topics.

As I said previously, the first step before tracking counters across nodes, is to use ETS table locally, to know the current count on topic, so obviously the value I get only concern users who join on that node.

Therefore if I deploy my app on 10 nodes, since the max is 70, I will get 700 users join on every single topic.

It’s not good for me because I don’t want those rooms to be too crowded. (I plan on using Phoenix Presences on those channels, which does not scale very well when there’s too many users)

1) First solution:
To solve that, my first thought was to divide the max, by the number of actives nodes:
For example on 10 nodes, max => 7 users

That means locally 7 users max can join a channel topic.

This works but it’s not perfect either because the users will be load balanced to different nodes.
They might fill room topics on certain nodes faster than on others, so it would create new topics with barely any users while the previous rooms are not filled completely yet.

2) Second solution

To avoid that problem, I could broadcast to every nodes when a specific room is full or since room id are created in ascending order, I could regularly inform the last id with available slots

This would occur in the room tracker after the aggregate.

The only issue is that, there’s always a delay between the moment where users are populated to a specific channel and the aggregate. That information would always arrive too late…

At this point, I\m running out of ideas

[LostKobrakai]

I guess the big question will be how hard of a constraint that “max 70 people” limit should be. You’re essentially running head on into the complexities of distributed computing with what you’re describing if you need this to be a hard constraint – as in neither disallowing someone to join when there’s less than 70 people as well as not allowing people to join from the moment there are 70 people.

@jaybe78

It’s not good for me because I don’t want those rooms to be too crowded. (I plan on using Phoenix Presences on those channels, which does not scale very well when there’s too many users)

I’d love to see any details for that and if that indeed is a problem with large individual channels vs. many users no matter the spread over channels.

[jaybe78]

Hello,

@LostKobrakai

as in neither disallowing someone to join when there’s less than 70 people as well as not allowing people to join from the moment there are 70 people.

The main constraint I have is to not have room overcrowded.
I set that limit to 70 for 2 reasons.

1. Performance: Some users talk about Presence timing out when it goes over 500

2. Those list of joined users will be displayed in a UI (mobile app)
   In the context of the app I’m building, I don’t need to return that many data.

On the other hand, yes I want those rooms to be as balanced as possible, because, still in the context of the application I’m building, those rooms are used to get users in touch.
So I don’t want an user to end up alone or only a few users to get in contact with (ideally).

What I should mention as well, is that I developed a background job “balancer” for the global counters (running every 20 seconds), which goes over the aggregated counters and broadcast to specific channels where to move when they become less crowded.

That works well as well, but the idea is to get those counters as balanced as possible as users are joining, rather than waiting for the balancer to do re-equilibrate the channels.

I’d love to see any details for that and if that indeed is a problem with large individual channels vs. many users no matter the spread over channels.

1 ) Are you saying that Phoenix Presence used on a channel of let’s say 10 000 users across multiple nodes would be the same as splitting those in 142 channels of 70 users or so ? (performance wise)

2)Let’s say for the sake argument that it does.

For my use case, I don’t need to return thousands of users to my UI, I only need a hundred or so.

Yes I could slice what Phoenix Presence returns but in the end, splitting those large number of users into small rooms allows to do some kind of “pagination”.

[LostKobrakai]

Your remarks make it sound like there’s no real reason for the number 70 exactly, so I’d suggest not going with a fixed cutoff point, but rather evalute a solution using low and high watermarks. Start creating new channels at a low watermark, but also make it so only the high watermark makes that number of users be a problem in a channel. That way if people are added to a channel after the low watermark is reached isn’t directly a problem. You can tune those numbers against expected latency of coordination between nodes as well as expected arrival rates of users. This makes even more sense if you seem to be moving people around anyways.

The part about people potentially landing in empty channels cannot really be avoided though. There will always be the case of all existing channels being considered full and the new person being put into a new channel. It’s however a matter of how likely you make that case. E.g. if your balancer does its job how likely is it for that case to happen.

@jaybe78

1 ) Are you saying that Phoenix Presence used on a channel of let’s say 10 000 users across multiple nodes would be the same as splitting those in 142 channels of 70 users or so ? (performance wise)

I have no idea about scaling persence, but I’m asking because both likely affect performance in distinct ways. There’s no single knob to dialing performance. I’d also argue that performance work is best evaluated by benchmarking your solution besides reading what others have to say.

[Phxie]

Is there a reason you’re using ETS instead of Redis?

You can use Phoenix Presence and push joins/leaves to Redis directly.

Here’s an example of using Redis to track users and user counts.

SADD/SREM handle adding/removing users to a set, while INCRBY/DECRBY update the count.

def handle_metas(topic, %{joins: joins, leaves: leaves}, _presences, state) do
  key_id    = "#{topic}:users"
  key_count = "#{topic}:count"

  commands =
    Enum.flat_map(joins,  fn {id, _} -> [["SADD", key_id, id], ["INCRBY", key_count, 1]] end) ++
    Enum.flat_map(leaves, fn {id, _} -> [["SREM", key_id, id], ["DECRBY", key_count, 1]] end)

  Task.start(fn -> Redix.pipeline(:redix, commands) end)

  {:ok, state}
end

If you want to assign users to room* topics, you could also use a Redis sorted set. Store room IDs as members and their user counts as scores:

ZADD rooms 65 room1
ZADD rooms 54 room2

You can then use ZRANGE or ZPOPMIN to find a room under the threshold (e.g., < 70) and assign users accordingly.

[jaybe78]

Hello

Is there a reason you’re using ETS instead of Redis?

I use ETS because it’s really fast in atomic operations.

I’m able to spread hundreds of thousands of users connecting at the same time within seconds among those topics.

if you want to assign users to room* topics, you could also use a Redis sorted set.
Well the balancer I developed already do that, I just use a gen server.

[Phxie]

If you want to use ETS that’s fine, I was just pointing out that Redis is designed to handle what you’re trying to do in a way that ETS and Genservers are not. Your trying to add cross node functionality to tools that are for local use, rather than use a tool that will work cross node by default.

@jaybe78

if you want to assign users to room* topics, you could also use a Redis sorted set.

Well the balancer I developed already do that, I just use a gen server.

If you were using a sorted set you wouldn’t be making this post because it solves the exact issue your post is about with regards to counting and distributing users to rooms atomically.

@jaybe78

This works but it’s not perfect either because the users will be load balanced to different nodes.
They might fill room topics on certain nodes faster than on others, so it would create new topics with barely any users while the previous rooms are not filled completely yet.

Again, you mentioned this above as well. With a sorted set you would just go down the list until you find a vacant spot and then fill it. If no vacant spots exist you create a new room. Sorted sets are ordered by highest counts.

I should also add, one of the reasons I’m pushing Redis so hard is that I was desgning something with ETS/Genservers a year or so ago for a couple of week before I hit a wall and finally tried Redis. Redis made everything 10x easier for me.

[jaybe78]

If you were using a sorted set you wouldn’t be making this post because it solves the exact issue your post is about with regards to counting and distributing users to rooms atomically.

So you saying that if I use redis I could get the current count for a channel across all nodes, while hundreds of thousands of users are joining ?

It would be as fast as ETS for read/update ?

[LostKobrakai]

@Phxie

I should also add, one of the reasons I’m pushing Redis so hard is that I was desgning something with ETS/Genservers a year or so ago for a couple of week before I hit a wall and finally tried Redis. Redis made everything 10x easier for me.

The big change here is not GenServer vs. Redis though. It’s centralized storage of data (CP) vs. distributed storage of data (eventual consistency / AP)

[jaybe78]

Yes that’s what I think, and I don’t see how I could have a performant algorithm if I have to get the updated global counter value from redis ?

I think what you said early makes sense.
I need to make some compromise regarding that max value, and probably accept that at certain moments, the counters won’t be balanced.

As long as my balancer does a good job, they will be eventually balanced

[Phxie]

@jaybe78

So you saying that if I use redis I could get the current count for a channel across all nodes, while hundreds of thousands of users are joining ?

Yes.

Redis can give you an atomic, cross‑node count while users join and leave in real time. It won’t match ETS’s raw speed, but building your own synchronization and network‑transfer logic for ETS updates erodes that advantage. At high join/leave volumes, custom distribution code risks race conditions and latency that Redis’s built‑in counters avoid.

You will never get 100%, always correct counts when dealing with hundreds of thousands of users connecting at the same time, but at least with Redis the results will be atomically consistent.

@jaybe78

It would be as fast as ETS for read/update ?

No.

ETS is local, it will always be faster than using an external source.

However, I should also add that although ETS is faster locally you still presumably have to manually process users/counts and send info between nodes. Your custom data distribution logic will not magically be done in an instant, especially at the scale of hundreds of thousands of users constantly joining and leaving.

I should add, your posts “problem” is the below. You have only mentioned speed whilst trying to counter Redis, but I recommended Redis because it answers your posts problem. You’ve changed the argument from “I can’t get the counts correct” to “but it’s gotta be fast”.

@jaybe78

It works quite well overall, but I have a “problem” which concern the total number of users on channel topics.

I’m not going to reply any further, as I feel I’ve expressed my opinion multiple times at this point and it would be a waste of time for me to continue to push a tool you obviously have no interest in using.

Apologies if I’ve annoyed you by turning this into ETS vs Redis, but I still maintain Redis is the correct tool for the job for track user presence with atomic room counts.

[jaybe78]

Don’t take it personal dude.

I haven’t said what you say is wrong.
I’m just asking questions.

Concerning Redis I thought about using it.
I use AWS, so I could use it but it’s really expensive.

I could deploy my own redis, but at this point I don’t feel like managing my own redis.
It would be extra work.

But yeah thanks for your inputs

[Phxie]

@LostKobrakai

The big change here is not GenServer vs. Redis though. It’s centralized storage of data (CP) vs. distributed storage of data (eventual consistency / AP)

I never said Redis vs Genservers though. I said ETS/Genservers, meaning the combination of ETS as a tracker/counter and Genservers as a method of passing the counts to wherever they need to go.

Redis does the job of both combined, just not as fast for local applications.

Also, do you know of any sites that track hundreds of thousands of users constantly joining/leaving rooms that don’t use an eventually consistent model?

Most sites will use a redis style tracker with a TTL heatbeat to display presence.

[garrison]

I’m a little confused with what’s going on in this thread. There’s an implication here that Redis is somehow different than using ETS tables, but there is no reason you can’t just put an ETS table on one node and use it exactly like you would use Redis! You would have no fault tolerance and higher latency, just like Redis. The only reason you would want to actually use Redis is if you wanted its vast array of fancy data structures, which in this case maybe you don’t, seeing as you have already implemented your system on ETS.

@Phxie

If you want to use ETS that’s fine, I was just pointing out that Redis is designed to handle what you’re trying to do in a way that ETS and Genservers are not. Your trying to add cross node functionality to tools that are for local use, rather than use a tool that will work cross node by default.

Again, it’s perfectly valid to use Redis for its functionality, but “ETS and GenServers” were absolutely designed to be used “cross-node”. OTP literally could not be more designed for that! It’s, like, the entire thing!

@Phxie

Also, do you know of any sites that track hundreds of thousands of users constantly joining/leaving rooms that don’t use an eventually consistent model?

This is not a literal answer, but the quintessential example would be something like an exchange/order-matching system: something which processes (financial) transactions. If you want to learn more about that the magic keywords are “LMAX architecture”, plus check out Tigerbeetle for a modern fault-tolerant rendition.

@LostKobrakai

The big change here is not GenServer vs. Redis though. It’s centralized storage of data (CP) vs. distributed storage of data (eventual consistency / AP)

Apologies in advance for nitpicking something you’re probably fully aware of, but CAP and “centralized/distributed” are separate concepts. You can have a distributed consistent system, and you can also have a single-threaded inconsistent system, though this is less common because in a single-threaded system consistency is so easy to achieve.

[garrison]

To the OP, I think you need to take a step back here.

What are the rooms for? What do they contain? Messages? State? Where is that information stored?

Do you need fault tolerance, or are the rooms ephemeral?

Assuming it’s not a big deal to “lose” a room, it’s okay to store each room’s state in one place, without worrying about replication or distributed consistency. Even just the user count is a piece of state.

Do you need to scale out? If not, you can just put all of the rooms in one ETS table on one node. Or in Postgres, or Redis, or whatever. That will probably scale into the millions.

If you do need to scale out, hash partition the rooms across nodes. When a user connects, choose a random node and ask that node to add the user to one of its rooms. The node knows the state of each room on that node, so it can pick one at random. Then if you need to look up any other state about that room, you can hash it and contact its node. If you need to handle nodes going down, use consistent hashing. Rendezvous is simpler than ring hashing.

When you pick a room to fill, finding the emptiest room is an expensive operation (there could be millions of rooms). Use the old load balancing trick: pick two at random and fill the emptier one. It’s near-optimal.

[Phxie]

I’m confused as to why you’re confused because I’ve explained my thoughts on the differences between ETS and Redis multiple times.

More importantly, this was not meant to be the ETS vs Redis debate it became.

OP posted an issue and said:

@jaybe78

At this point, I\m running out of ideas

So I suggested a new idea that I know can work for the problem they are having. I don’t get why everyone on this site feels the need to turn simple comments aimed at just offering an alternative insight into bizaare 1up quote competitions.

@garrison

The only reason you would want to actually use Redis is if you wanted its vast array of fancy data structures, which in this case maybe you don’t, seeing as you have already implemented your system on ETS.

actually

Your making out that I’m suggesting Redis just because of “fancy” data structures and that OP didn’t make a post stating their current system does not work the way they want it to. People are allowed to offer alternative solutions they think will solve the issue.

@garrison

This is not a literal answer, but the quintessential example would be something like an exchange/order-matching system: something which processes (financial) transactions. If you want to learn more about that the magic keywords are “LMAX architecture”, plus check out Tigerbeetle for a modern fault-tolerant rendition.

Whats the point in answering the question with something that has nothing to do with tracking user presence or the question being asked. The topic is real time presence tracking, not general transactions at scale.

Presence involves users constantly joining/leaving and moving to different channels which involves updating users, rooms and counts constantly. I had this same problem with you in another thread where you just kept taking things off on random tangents.

I’m done with this whole forum I think. I just wanted to share a potential solution to the issue in the post.

[jaybe78]

I’m actually here to hear different opinions… no one is holding the absolute truth on anything…

Stop taking things personally.
He has the right to disagree with you.

Sorry you feel that way.

[karlosmid]

Hi, maybe distributed cachex is an option to check:

Distributed Caches — Cachex v4.1.0

[Phxie]

@jaybe78

I’m actually here to hear different opinions… no one is holding the absolute truth on anything…

Stop taking things personally.
He has the right to disagree with you.

Sorry you feel that way.

You say no one is holding the absolute truth, but thats my critisism of the way he always responds to people.

The other day he was arguing with me because he has his own personal “correct” opinion of what the term “infinitescroll” means, that is different from the commonly understood term which resulted in a pointless discussion. He constantly argues with things you have never said, takes what you have said and either adds meaning to it, or tries to find meaning you never tried to exress and generally just nitpicks random things that are not relevant to the topic being discussed, or helpful in solving it.

Everyone has the right to disagree, but I have the right to get annoyed when people respond to me like this. The goal of a thread like this is to solve the main issue in the main post, not nitpick every little nuanced thing someone says or try to 1up people constantly.

I mean, what is the point of this? It has no relevance to anything being discussed in the thread at all but gets directed at me. What if people in the future come along looking for help solving the posts issue, then end up wasting time researching the things in his response?

@garrison

@Phxie

Also, do you know of any sites that track hundreds of thousands of users constantly joining/leaving rooms that don’t use an eventually consistent model?

This is not a literal answer, but the quintessential example would be something like an exchange/order-matching system: something which processes (financial) transactions. If you want to learn more about that the magic keywords are “LMAX architecture”, plus check out Tigerbeetle for a modern fault-tolerant rendition.

[garrison]

@Phxie

I mean, what is the point of this? It has no relevance to anything being discussed in the thread at all but gets directed at me.

I was simply trying to answer your question in earnest. Order matching is an example of a similar situation (orders join and leave the book) where strong consistency is required and the system effectively cannot be partitioned. If I had posted that question as you had I would have been interested in that response, so I gave it. If you were not interested, that’s perfectly fine; you could have simply ignored it!

@Phxie

The other day he was arguing with me because he has his own personal “correct” opinion of what the term “infinitescroll” means, that is different from the commonly understood term which resulted in a pointless discussion. He constantly argues with things you have never said, takes what you have said and either adds meaning to it, or tries to find meaning you never tried to exress and generally just nitpicks random things that are not relevant to the topic being discussed, or helpful in solving it.

Personal attacks aside (which I do not appreciate, regardless of who they’re aimed at), you are literally confusing me with someone else. That was not me.