Implementing distributed user counters

I guess the big question will be how hard of a constraint that “max 70 people” limit should be. You’re essentially running head on into the complexities of distributed computing with what you’re describing if you need this to be a hard constraint – as in neither disallowing someone to join when there’s less than 70 people as well as not allowing people to join from the moment there are 70 people.

I’d love to see any details for that and if that indeed is a problem with large individual channels vs. many users no matter the spread over channels.

Hello,

The main constraint I have is to not have room overcrowded.
I set that limit to 70 for 2 reasons.

1. Performance: Some users talk about Presence timing out when it goes over 500

2. Those list of joined users will be displayed in a UI (mobile app)
   In the context of the app I’m building, I don’t need to return that many data.

On the other hand, yes I want those rooms to be as balanced as possible, because, still in the context of the application I’m building, those rooms are used to get users in touch.
So I don’t want an user to end up alone or only a few users to get in contact with (ideally).

What I should mention as well, is that I developed a background job “balancer” for the global counters (running every 20 seconds), which goes over the aggregated counters and broadcast to specific channels where to move when they become less crowded.

That works well as well, but the idea is to get those counters as balanced as possible as users are joining, rather than waiting for the balancer to do re-equilibrate the channels.

I’d love to see any details for that and if that indeed is a problem with large individual channels vs. many users no matter the spread over channels.

1 ) Are you saying that Phoenix Presence used on a channel of let’s say 10 000 users across multiple nodes would be the same as splitting those in 142 channels of 70 users or so ? (performance wise)

2)Let’s say for the sake argument that it does.

For my use case, I don’t need to return thousands of users to my UI, I only need a hundred or so.

Yes I could slice what Phoenix Presence returns but in the end, splitting those large number of users into small rooms allows to do some kind of “pagination”.

Your remarks make it sound like there’s no real reason for the number 70 exactly, so I’d suggest not going with a fixed cutoff point, but rather evalute a solution using low and high watermarks. Start creating new channels at a low watermark, but also make it so only the high watermark makes that number of users be a problem in a channel. That way if people are added to a channel after the low watermark is reached isn’t directly a problem. You can tune those numbers against expected latency of coordination between nodes as well as expected arrival rates of users. This makes even more sense if you seem to be moving people around anyways.

The part about people potentially landing in empty channels cannot really be avoided though. There will always be the case of all existing channels being considered full and the new person being put into a new channel. It’s however a matter of how likely you make that case. E.g. if your balancer does its job how likely is it for that case to happen.

I have no idea about scaling persence, but I’m asking because both likely affect performance in distinct ways. There’s no single knob to dialing performance. I’d also argue that performance work is best evaluated by benchmarking your solution besides reading what others have to say.

Is there a reason you’re using ETS instead of Redis?

You can use Phoenix Presence and push joins/leaves to Redis directly.

Here’s an example of using Redis to track users and user counts.

SADD/SREM handle adding/removing users to a set, while INCRBY/DECRBY update the count.

def handle_metas(topic, %{joins: joins, leaves: leaves}, _presences, state) do
  key_id    = "#{topic}:users"
  key_count = "#{topic}:count"

  commands =
    Enum.flat_map(joins,  fn {id, _} -> [["SADD", key_id, id], ["INCRBY", key_count, 1]] end) ++
    Enum.flat_map(leaves, fn {id, _} -> [["SREM", key_id, id], ["DECRBY", key_count, 1]] end)

  Task.start(fn -> Redix.pipeline(:redix, commands) end)

  {:ok, state}
end

If you want to assign users to room* topics, you could also use a Redis sorted set. Store room IDs as members and their user counts as scores:

ZADD rooms 65 room1
ZADD rooms 54 room2

You can then use ZRANGE or ZPOPMIN to find a room under the threshold (e.g., < 70) and assign users accordingly.

Hello

Is there a reason you’re using ETS instead of Redis?

I use ETS because it’s really fast in atomic operations.

I’m able to spread hundreds of thousands of users connecting at the same time within seconds among those topics.

if you want to assign users to room* topics, you could also use a Redis sorted set.
Well the balancer I developed already do that, I just use a gen server.

If you want to use ETS that’s fine, I was just pointing out that Redis is designed to handle what you’re trying to do in a way that ETS and Genservers are not. Your trying to add cross node functionality to tools that are for local use, rather than use a tool that will work cross node by default.

If you were using a sorted set you wouldn’t be making this post because it solves the exact issue your post is about with regards to counting and distributing users to rooms atomically.

Again, you mentioned this above as well. With a sorted set you would just go down the list until you find a vacant spot and then fill it. If no vacant spots exist you create a new room. Sorted sets are ordered by highest counts.

I should also add, one of the reasons I’m pushing Redis so hard is that I was desgning something with ETS/Genservers a year or so ago for a couple of week before I hit a wall and finally tried Redis. Redis made everything 10x easier for me.

If you were using a sorted set you wouldn’t be making this post because it solves the exact issue your post is about with regards to counting and distributing users to rooms atomically.

So you saying that if I use redis I could get the current count for a channel across all nodes, while hundreds of thousands of users are joining ?

It would be as fast as ETS for read/update ?

The big change here is not GenServer vs. Redis though. It’s centralized storage of data (CP) vs. distributed storage of data (eventual consistency / AP)

Yes that’s what I think, and I don’t see how I could have a performant algorithm if I have to get the updated global counter value from redis ?

I think what you said early makes sense.
I need to make some compromise regarding that max value, and probably accept that at certain moments, the counters won’t be balanced.

As long as my balancer does a good job, they will be eventually balanced

Yes.

Redis can give you an atomic, cross‑node count while users join and leave in real time. It won’t match ETS’s raw speed, but building your own synchronization and network‑transfer logic for ETS updates erodes that advantage. At high join/leave volumes, custom distribution code risks race conditions and latency that Redis’s built‑in counters avoid.

You will never get 100%, always correct counts when dealing with hundreds of thousands of users connecting at the same time, but at least with Redis the results will be atomically consistent.

No.

ETS is local, it will always be faster than using an external source.

However, I should also add that although ETS is faster locally you still presumably have to manually process users/counts and send info between nodes. Your custom data distribution logic will not magically be done in an instant, especially at the scale of hundreds of thousands of users constantly joining and leaving.

I should add, your posts “problem” is the below. You have only mentioned speed whilst trying to counter Redis, but I recommended Redis because it answers your posts problem. You’ve changed the argument from “I can’t get the counts correct” to “but it’s gotta be fast”.

I’m not going to reply any further, as I feel I’ve expressed my opinion multiple times at this point and it would be a waste of time for me to continue to push a tool you obviously have no interest in using.

Apologies if I’ve annoyed you by turning this into ETS vs Redis, but I still maintain Redis is the correct tool for the job for track user presence with atomic room counts.