Mnesia vs Cassandra (vs CouchDB vs ...) - your thoughts?

I think it’s moving to “ForgETS” or something like that which is their reimplementations of mnesia on top of ets.

I’d use scylladb for storing unimportant data like messages.

I particularly liked this talk about scylla https://www.youtube.com/watch?v=4BVdZw9uUMA

But I haven’t used it in production yet though.

I don’t think anything in particular has changed over the year in regards to mnesia.

Things you need to know:

• Although WhatsApp use mnesia they don’t use it in the “normal” way. They do async dirty transactions and they have their own replication between nodes (they have two nodes per partition). It is also used for temporary type of data.
• Keep in mind, that unless you go with one of the new mnesia backends (such as leveldb) you must hold all your data in memory.
• mnesia supports transactions (cassandra has light-weight transactions)
• They are distributed but they do differ how this is done. Cassandra is more (and easier) scalable than mnesia.

Overall, if your data is more ephemeral in nature mnesia is a good choice, you need transactions. If you have large amount of data and the need to scale a lot cassandra would be better.

You have big company behind Cassandra https://www.datastax.com/ with theirs tools, Apache Sorl integration (search engine)
Also Casandra has good integration with big data eco system like Apache Spark, Apache Kafka.

No, mnesia has always had the ability to store data on disk. You just specify on which nodes you want disc copies. The only earlier limitation has been that the disk files must not be larger than 2 Gb but you got around this by partitioning your tables.

Yes, you are right. For small datasets it is doable. On the other hand if they are comparing it with cassandra I just assumed they had “a lot” of data.

In my experience the dets backend is not feasible. The small amounts of data it can keep and the overhead of managing table partitioning is not really worth it if you have large amounts of data. Just assume you have a dataset of 1TB, that in itself means 500 partitions and to cater for growth you want more than that. And because it performs best when you add partitions in the power of two the next step is 1024 partitions which you need to manage. The time it takes to change the number of partitions is also not very optimized. The process uses temporary ETS tables of bag or duplicated bag with an O(N) behaviour. If you have more than 10-20K keys it will take forever to add a single fragment. (actually I have not tested this with dets, only the disc_copies backend so if it differs I may be wrong here)

Yes, it very much depends on how much data you want manage.

A third option I have came across is to use CouchDB, which also some large parties use. It’s also built on the BEAM, but unfortunately currently does not support running in the same BEAM as your app. The main difference between CouchDB, Mnesia and Cassandra are:

Cassandra

• runs on the JVM (There is also a drop-in replacement called ‘Scylla’ that runs on C++).
• It stores data in a relational format (tables with specified columns with specified types)
• uses SQL(-like; some features are not supported because of the distributed nature) statements to query data.
• Updates are per-column, so conflicts are also per-column (although lightweight transactions to ensure all-or-nothing updates are also supported).
• For cassandra to work well, your server’s clocks need to be synchronized.

CouchDB

• runs on the BEAM. (but including it inside your BEAM instance is not supported)
• It stores data in a document format,
• javascript map/reduce functions to query data.
• conflicts are resolved by you yourself periodically fetching multiple revisions and storing a new conflict-free version; until you run this, last-write-wins.

Mnesia

• Runs inside your BEAM.
• It stores data in a set/ordered_set/bag of Elixir/Erlang datatypes.
• Erlang QLC queries to perform queries.
• Does not handle conflicts; although libraries such as unsplit and reunion allow you to do this by writing your own conflict-resolution function.

According to the Mnesia documentation, the 2GB limit only applies to disc_only_copies; ram_copies and disc_copies are only affected by the size of available RAM. For practical purposes this still means that >16GB (or maybe >32GB or >64GB, but then we definitely are at the limit) tables need to be partitioned.

To be honest, I still have no clue which database engine I’d like to pick. I might end up going for the one that restricts my system the least for the future because I see no clear disadvantages from picking one over the other.

I start with Mnesia because its just there. No extra installs needed, no mucking with another language (i.e. SQL) to interact with the dbase. Most of the systems I work on don’t do Updates on a row, only Creates so the whole conflict resolution typically don’t come into play.

For longer term storage, past a week in my cases, I would move the data from Mnesia to a RDMS or another type of database. Typically which ever flavor the Sales/Accounts/Finance people are keen on using.

Going with the last few posts, put it all behind a module interface, start with the most basic, which might just be ETS or so in RAM, maybe mnesia later, maybe move to eleveldb or something later, maybe even migrate to PostgreSQL later, all without changing the interface.

I saw your request on the couchdb mailing list, and while it’s “not supported” it’s not very hard to do it either, as this is actually how CouchDB is built internally anyway.

These notes are for 1.7.x which is what I’ve got to hand:

• create your elixir application as usual and compile it
• unpack the release into couchdb plugins dir (or /usr/local/lib/couchdb/erlang/lib/... ) for whatever your OS uses
• add a [daemons] section to your local.ini config with myapp={'Elixir.MyApp', start_link, []} which will ensure it gets started up as part of the main supervision tree of couch
• set ERL_FLAGS environment variable to use -config /usr/local/etc/couchdb/sys.config to get your custom application settings loaded up
• look through https://github.com/apache/couchdb/tree/master/src/couch_plugins which has some useful comments and source snippets

I’m reasonably sure this is complete, and while the complexities of 2.x series with cluster support make this moire complicated, the guts of that should remain the same.

At some point, somebody will decide to put a nice tidy shim in between the chttpd “clustered httpd” layer that exposes the soft erlang underbelly of couchdb, and then you’d have native erlang terms accessible to Elixir as well. There are a few not-so-handwavey details to deal with - no maps yet as couch uses proplists (no maps at the time in Erlang, and to support some quirks of JSON), things like structs have no way of being converted back to JSON at the http layer, and a bunch of valid BEAM types that would need to be cleaned when turning them into JSON again etc etc, once the shim is available.

The big issue is that releases for a clustered db are a serious thing, and we tend to update and restart our apps far more often than our DBs. If you’re already co-locating the DB and the app on the same server, only the JSON encoding & recoding is the bottleneck.