Do you think random would be meaningfully faster than hashing? I kinda doubt it would make a substantial difference.
Either way the correct approach IMO would be to load-balance with the two random choices trick because randomness alone will randomly overload random nodes 
I don’t know, I’d have to measure it, most likely random would be faster. Although that’s not the point of Mesh, mesh is basically a service discovery with the added benefit of being able to activate and monitor the target (process). We’re not talking about a pool, nor endpoints (servers), nor consensus systems; in Mesh we’re only talking about affinity groups. Where knowing the logical Process name, not the PID, nor a registry query, nor anything like that, you are able to send a request, and you are able to do this with “node affinity”, that is, within a larger system (cluster) you can have subsystems and you can logically divide processes among these subsystems.
The reason why random sampling wouldn’t be suitable here is that, in this use case, given that the cluster doesn’t change much, you’ll always be communicating with the same target, and if the cluster eventually changes, it will likely be synchronized at some point.
We will still be working on name collisions, better handling of various issues, the possibility for users to implement their own strategies, and so on… This is just version 0.1.1 of the library, which we are sharing early precisely so we can gather valuable feedback like the kind you gave us.
Without any pretension, I’m sharing the result of one of our benchmarks:
Creating 40,000 actors (10000 per capability)
Time: 2.12s
Throughput: 18827.1 actors/s
Success: 40000/40000 | Failures: 0
Actor distribution per node:
bench@127.0.0.1: 10000 actors
balanced_node_1@127.0.0.1: 10000 actors
balanced_node_2@127.0.0.1: 10000 actors
balanced_node_3@127.0.0.1: 10000 actors
50,000 invocations on existing actors
Time: 3.23s
Throughput: 15484.83 req/s
Success: 50000/50000
Hash ring distribution for 10,000 samples
Expected distribution per node:
bench@127.0.0.1: 2500 actors (25.0%)
balanced_node_1@127.0.0.1: 2500 actors (25.0%)
balanced_node_2@127.0.0.1: 2500 actors (25.0%)
balanced_node_3@127.0.0.1: 2500 actors (25.0%)
Standard deviation: 0.0
FINAL CLUSTER STATE
bench@127.0.0.1:
Status: Online
Processes: 14237
Memory: 124.83 MB
Actors: 10000
balanced_node_1@127.0.0.1:
Status: Online
Processes: 14213
Memory: 101.74 MB
Actors: 10000
balanced_node_2@127.0.0.1:
Status: Online
Processes: 14213
Memory: 104.38 MB
Actors: 10000
balanced_node_3@127.0.0.1:
Status: Online
Processes: 14213
Memory: 103.04 MB
Actors: 10000
Total actors in cluster: 40000
Aloha – thanks for sharing this; it aligns with my own plans.
I opened issue #2 and PR #3: Fix shard lifecycle cleanup, monitor recovery, and capability isolation by nshkrdotcom · Pull Request #3 · eigr/mesh · GitHub.
Full disclosure: I set the investigation criteria and verified the results; Codex and Claude Code handled the investigation, failing tests, fixes, and the issue/PR/commit text.
Not meaningfully, but why do hashing by some when there’s no requirement to route things into the same node in the first place?
With all respect, this is a hash based routing, based on the key provided by the user. There is a huge chance that some nodes will never be selected if user provides less keys than there are nodes (for example). But even if not, there is guarantee that user provides values which return more or less plain distribution of hashes.
For example, it routes based on request.id which is a string set by user. There is not uniqueness check on it and it can be just hardcoded (or set to nil by default)
If the user has defined incorrect inputs, they shouldn’t expect guarantees. What we can do is warn them that something is being done incorrectly. We’ll add this validation, and thank you again for the suggestion.
But the goal itself is for them to provide an identity; they do this because they know they’ve defined a process that represents something they expect to happen. It’s not the goal of this library to provide guarantees about the user’s business rules. The user should build that on top of this library, in their own codebase. But as long as they provide a valid identity, they will be directed to the process the user defined.
You can implement a custom strategy and implement this behavior if you wish; the most recent documentation explains how to do this.
I’ve read the code and here’s my review
This is a poor library with a lot of bugs, misleading documentation and poor applicability.
It essentially does 2 things.
Mesh.register_capabilities([:game, :chat])
GenServer.call with specified payload{:ok, pid, response} = Mesh.call(%Mesh.Request{
module: MyApp.GameActor,
id: "player_123",
payload: %{action: "move"},
capability: :game
})
shard_count of processes per each capability called “actors” in terms of this library (but I call them just shards here) and this request is routed to one of these shards. That means if the node has two capabilities and max shard count is 10, it is going to maintain 20 processes (all of them are lazily initialized).Capabilities is a distributed data, and as all such data it is subject to CAP. Current solution is not qualifying for any of those letters and, putting CAP aside, provides very poor guarantees. If other node can’t start the Mesh and set the capabilities in time, the current node won’t know about it. It uses replication on net_kernel.monitor_nodes events, which just tries to do asynchorous rpc to get the capabilities of other node and share current capabilities with it.
That means that any change to cluster configuration or groups is eventually consistent (at best) and it is quite possible that your request would get routed to the already dead node. If so, this library provides no fallback and you’d just have a dead request
Alright, let’s forget about previous paragraph and imagine that every node has the exactly the same information about the groups (aka “capabilities”).
In pseudo-code it looks like this
shard_id = hash(request.id, shard_count)
nodes_in_group = Enum.sort(nodes_for_group(request.node))
node_to_execute_in = Enum.at(nodes_in_group, rem(shard_id, length(nodes_in_group)))
:rpc.call(node_to_execute_in, fn ->
shard_owner = via(shard_id) # Yes, shard_id, the same as on the first line
GenServer.call(shard_owner, {:call, request})
end)
# Where the GenServer.call executes this
pid = start_or_get_shard_process(process, shard_id) # Again, shard_id
response = GenServer.call(pid, request.payload)
{:reply, response, ...}
And it has many problems. First thing is that routing to the node is not consistent. Second is that shard within node is selected twice, which is a bug.
Then, and it is the most beautiful bug here, shard_id is used as a key for selecting a node in a group and the shard process. That means, that if I have two nodes and 20 shards on each node, only 10 shards on each would receive the request, which is very fun. In general, if I have N nodes, only shards / N amount of shard will receive the request.
Because if I hit the first node on the list of two nodes, that maens that shard_id is dividable by two, and all shards with shard_id not dividable by two won’t receive a request on this node ever.
And for problems which I couldn’t reflect in the pseudo code:
GenServer.start_link(Module1, ...), and if I then send a request which would end up in the same shard, but with Module2 passed as the implementation, it would still hit the shard created by the first request, with Module1ActorTable process on each node which just does nothing.PartitionSupervisorMesh.Cluster.Membership schedules monitoring after 100ms, while it can just do it in handle_continueglobal for locking on only the current nodeUnclear use-case, bad implementation, misleading documentation.
Same functionality (but without a lot of bugs) can be achieved by using any other process group solution (including bultin global_group, pg and third-party Horde, Swarm, ProcessHub?, syn).
But given that shard processes are expected to be stateless and rounting to be inconsistent, it all boils down to just two lines
node = Enum.random(Node.list())
:erpc.call(node, fn -> ... end)
I’ve just reread my post, and it sounds a bit toxic. I want to make clear that given that this implementation is poor, it is a good start and the whole idea is not strange, but it is a classic problem which was solved several times by other authors from Elixir community and what you’re doing with Mesh is a good starting point to explore the problem space
My personal recommendation would be to
Horde library solves the similar problem, but it uses eventually consistent delta-CRDT merkle tree structure to (eventually) maintain the same version of the information about which processes belong to what groups. Other good example is ProcessHub which is, afaik, still in active development phase.I am looking forward to seeing the new version of Mesh which would use some interesting distributed algorithm. Happy coding 
Can you articulate your intended usage scenario with a concrete example? Do you want “node affinity” based on performance concerns, ie. hot cache, or physical attributes of the nodes ie, big memory vs small memory, or security/regulation concerns ie. to make private enclaves within a public cloud?