Replicated Cache/DB in elixir/erlang? What's the best solution? Use case inside

I am not sure if @sb8244 handles dynamic nodes added or removed from the cluster:

Same caveat as above, but do not think these 2 libs handle replication out of the box:

https://whitfin.io/cachex-v3-1-and-the-return-of-distributed-caches/

https://medium.com/erlang-battleground/distributed-caching-in-elixir-using-nebulex-9af589186caa

If you have a limited number of nodes, the nodes don’t change that often, it’s really just a cache and you don’t mind lost writes, and you have a limited amount of data to replicate then you can probably do something naive. The most basic would be to listen for node up and down events. when you detect a new node has joined you can replicate your data to them. You can probably use pg2 to maintain a registry of cache processes. You can replicate to those processes and have them write into an ets table or something similar. This approach is very naive and overly chatty. Plus it’ll definitely result in lost writes eventually. But it may be ok for your use case.

If any of those caveats are deal breakers then you may want to use an external process like redis or memcached or pull in a more complete solution like lasp or erleans. Otherwise you’re gunna find yourself solving some pretty gnarly distributed systems problems .

This solution I presented is definitely on the naive side. The advantage of the is that it’s easy to grok and debug, but Chris’s points are very true.

I found that the more complex tools were too much for my simple cache case and went with pg2 + cachex. This does handle dynamic node membership because the cache process sends out a request for cache load when it starts. Lots of room for improvement but a decent starting point for the naive solution.

So you only want to use this as an ephemeral cache?

It might be worth looking into Phoenix Prescence, because it also fills in this need somewhat.

That feels like it would be a misuse of Phoenix.Tracker (you wouldn’t want presence in this case). Tracker is great for monitoring processes across the cluster and can accept metadata, but isn’t the best choice if you don’t need process tracking.

Presence uses CRDT which are available standalone as well, although they can have performance trade-offs so make sure to throw scenarios at it before committing.

I was following the work being done on FireNest (https://github.com/phoenixframework/firenest/pull/21) - with a branch having some work on seemingly generic state replication. Unfortunately that project has now been archived. Hopefully we’ll see some movement in the replication arena from the core team again.

From the latest commit there:

Work on firenest has been halted. We plan to incorporate some of those features in Phoenix.PubSub instead.

hi, any new experience with this issue after 2 years? Or ideas about good solution for this kind of use case? Have you considered Mnesia? For example Memento is Elixir wrapper around Mnesia.
Did you use Cachex? Any feedback?
I have similar issue and I’m trying to get feedbacks/experience from users of these solutions.

FWIW, I used the Cachex + pg2 solution and had 0 issues with it in production since it shipped. I’ve since left that company, but my understanding is that it’s just churning along doing its thing.

So that’s a good endorsement for me, although it depends on your use case.

To be fair I’d just go for memcached or Redis and forget about it. They both serve this goal excellently.

If there is no distribution involved I’ll go 100% Elixir but when it gets to 3 or more machines my priorities change.