I was trying to address a distributed genserver, which is ensured to be unique by the Horde.DynamicSupervisor supervision tree.
I can check this with Supervisor.which_children - showing me local and remote pids.
I can derive recursively which_children to child supervisors of Horde and so address process remotelly.
But the Register should also do it.
“A distributed process registry.”
“Local changes to the registry will automatically be synced to other nodes in the cluster.” https://hexdocs.pm/horde/Horde.Registry.html
But reading the code of register/lookup
It’s working over :ets.
Not only this but I can use Horde.Registry to register the same key over 3 nodes.
Not only this. For each node, Horde.Registry shows me different local pids.
Both Horde.DynamicSupervisor, and Horde.Registry on my application children list.
Am I missing something? Because :ets is a local resource. And even if the Registry is said to be distributed and it sure have deltacrdt configured on start.
I could not validate distributed behavior.
Make your own assessement if you believe it to be production ready.
It is unclear what problem you’re trying to solve but you should not look past the robust Erlang pg registry. There is also a distributed supervisor based on Erlang pg called pogo. It is likely to be a more robust foundation to base a solution on.
For establishing your cluster lib_cluster is excellent and offers a range of discovery methods.
If you have a large geographically dispersed cluster and need greater control over distribution there is also partisan.
Nothing stopping you from storing a remote pid in an ets tables. You just can’t directly access contents across the cluster. You can deposit the same pid in multiple ets tables under the same key to ensure availability.
A few things to remember:
IIRC pg uses two phase commit. It might be slow and I’m not entirely sure the soundness has been proven.
horde and swarm use CRDTs to back state. So note that your cluster can be inconsistent if there is communication lag. The consistency paradigm is eventual
Tread carefully!
If I had a wishlist, a distributed supervisor based on VSR would be it (raft has serious problems)
You may be referring to the old slow pg that pg2 replaced.
The docs for the ‘new’ pg in OTP/23 that deprecated pg2 explicitly states that pg is eventually consistent. The source code for pg indicates it was contributed by WhatsApp.
pg doesn’t have a long list of critical crashing problems unlike some other options mentioned here.
In theory, theory and practice are the same. In practice they are not.
I have been working on a comparable library ProcessHub. Although it’s still in the alpha stage, active development is ongoing.
I’m not going to mention the other libraries already pointed out but alternatively, you may want to explore Swarm.
While you could develop your implementation using the pg module, which is robust and would perform well in your described scenario, you may encounter challenges when the distribution model becomes more complex.
I think maybe you’re confusing the difference between cluster formation and distributed registries.
Horde is a distributed registry and relies on a clustering implementation to actually get nodes talking to each other, hence the mention of set_members. I’m guessing you need a cluster configured. I’ve run Horde in production and if your cluster is configured and the registries are listening then even if you somehow were to double register on separate nodes for the same name before it was synced, one would get killed once it did sync. If you’re not seeing one process get killed for a name conflict then your horde registries are not getting node cluster updates.
As I mentioned before, I really do have a cluster formed in a k8s cluster using libcluster.
I really can see process distribution of horde working very well. But the registry does not.
I spawn in my 3 nodes the same process with the same key in my 3 nodes, and Horde.Registry gives me a different pid for each node. AND does not know the other nodes keys.
Hello, I’m the author and sole maintainer of Horde.
Just popping in to answer a few of your questions:
Is Horde Registry really distributed?
Yes, it is backed by DeltaCRDT, which has leaderless syncing. Updates that come in are written to an ets table for quick read access. Writes go through the CRDT.
Am I missing something?
It looks like you have configured Horde.NodeListener for your Horde.DynamicSupervisor, but not for your Horde.Registry. I checked, and the guides are not super clear on this, probably owing to the order in which things were built (Horde.NodeListener was added after the guides were written).
Others have mentioned this already, perhaps you should re-read their responses.
Unmaintained
Horde is a library that, as I have learned, has some limitations, but still remains useful to quite a few people (>9k downloads last 7 days). I don’t have the time nor the interest in large development efforts for the project, and there are not a lot of new PRs coming in to the project.
I try to stay on top of things, but the obvious bugs have been ironed out a long time ago, and aside from a few small issues, I would say that Horde is doing fine.
Finally, I will note that Horde is provided free of charge and without any kind of guarantee that it will work.
