How to use ETS as a global cache?

It’s possible. You could always build a http endpoint around ets and server http requests that way. But none of that comes “built in” and I doubt it’s worth it in the general case.

On the beam you also have mnesia which allows you to share kind of an ets table across the cluster, but it’s not quite straight forward to use. That’s local state synchronized across the cluster and not accessing one shared resource.

Without much detail on why you need a global cache I’d probably suggest staying with redis, lacking any reasonable means of doing a tradeoff with other options.

yes, for sure.

If machine A and machine B are both elixir machines, you could use either Cachex or Nebulex, both takes advantage of a cluster of machines to distribute ets tables across nodes

I particularly think that if you have an elixir application is more simple and less expensive clusterizing your nodes and use ets than use redis.

If you don’t want to bring an external dep in, the BEAM ships with Mnesia which sits ontop of ETS and adds a consistent distributed database.

However, when two nodes stop communicating due to a network fault Mnesia can be tricky to recover.

This is what http_cache_store_disk and http_cache_store_memory do when cluster_enabled is set to true: they exchange cached responses by using distributed erlang and store them either in ETS table or on disk (and in this latter case, metadata is still stored in memory).

They take into account that nodes can have different requirements such as available disk space or memory and therefore they handle cached response autonomously. A cached response can be discarded from one node and still be available on another.

If you want to cache HTTP responses from Phoenix / Plug, you can take a look at plug_http_cache.

Is fetching a cached value from another node really worth it? Is it crushingly faster?

I have my doubts. Would love to see actual metrics.

So your hypothesis is that a global Redis instance will always be faster than a Cachex cluster, correct?
I am not sure how the cluster works, but if data is replicated in all the cluster machines, the speed should be faster, although you now have other problem to deal with, such as the split brain.

What about running separate caches on both instances? You will waste a little bit more ram, however you will get rid of a lot of complexity, which IMO is perfect, as ram is dirt cheap these days.

No, my hypothesis is that having a separate local cache for each node is going to work best. People really over-optimize for the wrong things I feel (and some of my practice has proven this hypothesis correct; accentuating the “some” here). Having 3 separate nodes doing a DB query cache each is honestly not a big deal at all, especially if the cached DB query cache should last at least 10 seconds; even in these conditions a local cache is more than enough.

To me the whole idea of “fetch cache from the network” is just hilarious in general, even though I’ve witnessed cases where it was still worth it (we’re talking results from SQL materialized views that were taking 10+ seconds to calculate; in all other cases however, distributed cache is just technology triumphing over itself, and over common sense as well).

No need to fetch from the network upon user request if each node has its own local cache, and preemptively fills its cache with data from other nodes The only role of a cache, after all, is to be full, and with the “hottest” objects (HTTP responses, …).

This is what http_cache_store_* do when clustering is enabled: each node aggressively downloads HTTP response it doesn’t know to become full quicker and have the latest objects. Nodes also query 1_000 objects (by default) from other nodes at startup to warm up.

So how does that work exactly? At app startup one node warms up its cache and all other nodes download that warmed cache from it before they start?