Replicated Cache/DB in elixir/erlang? What's the best solution? Use case inside

Redis is rock-solid, what problems did you have with it?

redis and other tools are good, but cloud service where we have to run it have some issues. K8s works, let’s say works without huge issues. But other services has outages. And running own service in k8s is first expensive because you have to have bigger resources for k8s itself and it still doesn’t resolve whole issue.

Only one reason why we use k8s, we have 2 envs(prod, non-prod) and together we runs about 30-40 elixir apps there. So we use it only because of orchestrations. We use flux so gitops resolve a lot of issues for us. But I really hate it. We used docker swarm 4 years ago and it was bad. I’ve never used Nomad, so not sure about it. We don’t have own SysOps department so we use managed solution from IBM. IBM is only one allowed cloud provider by our regulator. We can order some SysOps company/contractor to create some infra but it has to be maintained and we don’t want to care about it. So looks like we don’t have another options. We can not use AWS btw.

I agree that in-process cache (:ets) is the super-performant solution. But I’m not sure how good the BEAM is if you want to have a distributed ETS cache. Hence why I proposed other programs.

Redis is rock-solid, what problems did you have with it?

Additionally, be extremely careful with k8s. If you don’t have 10+ different apps and storage services with management of private networks and special-tailored infrastructure then the chances are 99% that you don’t need k8s.

You would likely be better served by Nomad or Docker Swarm.

I’m talking about 4-6 apps, pods in k8s. Yes, 3rd party/external service for it good solution. But it has to be fast and also we have issues with these external services. For example my use case is about loosing dependency on DB(Postgres) so we can be fully (most important features) available even when DB is not available. We are preparing for new client who will use api and some web endpoints with rate about 300 r/s + other clients. So downtime a few seconds can be huge problem for us. But I don’t have budget to get some huge solution for it. Even, I don’t think I need black box solutions when we have power of BEAM.

To be fair I’d just go for memcached or Redis and forget about it. They both serve this goal excellently.

If there is no distribution involved I’ll go 100% Elixir but when it gets to 3 or more machines my priorities change.

FWIW, I used the Cachex + pg2 solution and had 0 issues with it in production since it shipped. I’ve since left that company, but my understanding is that it’s just churning along doing its thing.

So that’s a good endorsement for me, although it depends on your use case.

hi, any new experience with this issue after 2 years? Or ideas about good solution for this kind of use case? Have you considered Mnesia? For example Memento is Elixir wrapper around Mnesia.
Did you use Cachex? Any feedback?
I have similar issue and I’m trying to get feedbacks/experience from users of these solutions.

From the latest commit there:

Work on firenest has been halted. We plan to incorporate some of those features in Phoenix.PubSub instead.

I was following the work being done on FireNest (https://github.com/phoenixframework/firenest/pull/21) - with a branch having some work on seemingly generic state replication. Unfortunately that project has now been archived. Hopefully we’ll see some movement in the replication arena from the core team again.

That feels like it would be a misuse of Phoenix.Tracker (you wouldn’t want presence in this case). Tracker is great for monitoring processes across the cluster and can accept metadata, but isn’t the best choice if you don’t need process tracking.

Presence uses CRDT which are available standalone as well, although they can have performance trade-offs so make sure to throw scenarios at it before committing.

So you only want to use this as an ephemeral cache?

It might be worth looking into Phoenix Prescence, because it also fills in this need somewhat.