So… we’re a lot further in development right now.
Currently, we’re using Mnesia, mostly because it allows us to postpone the choice for another datastore for later.
At first we thought Mnesia might be useful to use as a distributed datastore that would scale to millions of users all on its own, but:
-
The number of nodes in a database cluser is not 1:1 with the number of user-serving application nodes. Separating these into two layers would make the reason to let your database live ‘in’ your application no longer be worth it.
-
Messages in our chat application are broadcasted to all users connected to a chat channel (== a phoenix channel) in parallel to storing something to the datastore: In effect, the datastore is only used to read back in history when (re-)connecting to a channel at a later time. So: Read speed (the extra bit you get from having the database live inside your appiication) is not thát important here.
-
Mnesia does not do many of the things that are important for a large distributed datastore system on its own: It would mean building a lot of custom logic to handle the following features that you’d like in a distributed, fault-tolerant datastore:
-
network partitions resolving. (Mnesia suffers from split-brain, and performing read-time fixes is non-trivial)
-
proper replication/fault tolerance,
-
dataset sharding/consistent hashing etc. (Mnesia has limits to table sizes; table fragmentation is supported, but a very leaky abstraction when used directly; and you have to manually manage the fragments (and e.g. resizing them when new nodes are added)).
-
run-time network topology reshaping (mnesia is completely configurable at runtime, but does not have any strategies on its own).
So, together, these reasons are plentiful enough to keep our eyes open. There’s a high possibility that we’ll switch to either Riak or Cassandra or CouchDB. However:
- Riak is cool, has great features and interacting with the Erlang-based system feels relatively natural from Elixir. However, Basho (that made Riak) is no longer functioning. What does/will this mean for the longevity of the database, and support etc?
- Cassandra is supported by a large group of companies, and is e.g. what Discord currently uses in their Chat system (blog post). However, it’s based in Java, and its last-write-wins methodology of handling conflicts (which is non-optional, there is no way to manually resolve conflicts or e.g. build CRDTs) is frequently critiqued (including in Discord’s blog post).
I’ll look into CouchDB more before comparing it with these two again. Anyhow, in the meantime I’d like to hear if anyone has used these systems recently and has some more feedback: Almost all critiques you can find on the internet are multiple years old, and especially now that e.g. Basho is no longer a functioning company, the landscape might have changed somewhat.
Food for thought.
