Thank you for following up, here’s my take that has proven unpopular at times:

…We simply gave up the idea of a distributed cache altogether. To me and the several teams I’ve been a part of that needed this (or more accurately: believed they needed it) it turned out to be simply wasteful; we viewed it as replacing one DB with another so we just opted for per-node caches. Solutions varied, between naked ETS, to cachex, to ane, and even to our own Agents / GenServers holding on to specific data states.

Interestingly enough, this almost did not change our stats of cache hits / cache misses. It did at places but there we opted for pre-warming caches on each node (based on statistics about which keys were hit the most) and then we practically could not find difference in the hit / miss ratios pre-Redis and post-Redis.

So if you were to ask me if I was your consultant, 99% of the time I’d tell you “You don’t need a distributed cache, use in-process per-node caches”.

Disclaimer: I worked on projects with substantial traffic but not on the scale of, say, Amazon or Walmart. At those scales there the actual single-source-of-truth caches probably make sense. But I found that for everything below that they don’t.

i’d really avoid riak-core-lite if partition is not a requirement for you. for cachex and nebulex you don’t need rolling deployment if you’re replicating the entire cache in all nodes.
afaik you only gonna need rolling deployment if you’re dealing with partitions, bc every new node gonna require that the partitions redistribute data between them and that is a slower process in comparison with just dumping the entire ets table to another node.

Apologies for lack of clarity, I wouldn’t use Scylla. I’m just referencing how they opted for raft for consistency/performance instead of the gossip/paxos implementation cassandra uses. For greater context, scylla is a rewrite of cassandra in C++. I was just mentioning how I’m tempted to develop an elixir solution that tries to get more performance using a different consensus algorithm. Also, this isn’t mission critical. This is more for fun and seeing what other folks do or are experimenting with.

Khepri looks interesting, thanks for the share. Also, agreed separation of concerns in a cluster can further increase performance. Thanks for the feedback.