Hi, sorry for taking so long to respond, Horde’s backing store (DeltaCRDT) is eventually-consistent, and there is no mechanism for checking to see if the locally-applied changes have been synced.

Using Horde.Registry.put_meta/3 during shutdown is unlikely to work well, as @hubertlepicki has pointed out, since syncing is performed asynchronously, and since you’re in the process of shutting down the node, it’s highly likely that the node will stop before any syncing can happen.

My recommendation for now, as Horde is not persisted, would be to only use Horde for data that you can afford to lose. Process state handoff would be better implemented using a database like Postgres or Redis, or perhaps something else.

I strongly suspect what is happening here is that you’re shutting down the whole node before Horde.Registry manages to propagate its state across other nodes in the cluster. CRDTs are eventually consistent and it’s not an immediate changes that you’re putting onto registry, they have to be synced back to the rest of the cluster.

I never did what you’re trying to do, with using Horde to pass the state to itself so not sure how to handle it and if it’s good idea at all.

What I found a reliable way to pass the state in the above situation, and also to ensure that a required process indeed starts on the other nodes was to use PostgreSQL to store the state. In my case there were two scenarios: one in which I scheduled an Oban job to start a corresponding process on other nodes, and in it’s payload I was passing the state, in another case I was just saving the state to a database and process would be started on-demand on other nodes when required and would pick it up.

I believe using Process.sleep is not a very reliable option since the distribution of state may take longer or less due to the underlying network. I typically address similar cases using message passing.