Best way to start Mnesia in Elixir/Phoenix application

As per the docs you need to stop Mnesia in order to create the schema:

mnesia:create_schema(NodeList) initializes a new, empty schema. This is a mandatory requirement before Mnesia can be started. Mnesia is a truly distributed DBMS and the schema is a system table that is replicated on all nodes in a Mnesia system. This function fails if a schema is already present on any of the nodes in NodeList. The function requires Mnesia to be stopped on the all db_nodes contained in parameter NodeList. Applications call this function only once, as it is usually a one-time activity to initialize a new database.

When starting Mnesia you must wait for all the tables to be ready, like it is said in the docs:

Table initialization is asynchronous. The function call mnesia:start() returns the atom ok and then starts to initialize the different tables. Depending on the size of the database, this can take some time, and the application programmer must wait for the tables that the application needs before they can be used. This is achieved by using the function mnesia:wait_for_tables(TabList, Timeout), which suspends the caller until all tables specified in TabList are properly initiated.

A word of caution:

• I lost all the records in my database when starting Mnesia after stopping it with ctrl + c + c and I was not able yet to find the cause for it, neither replicate it. While I am using Memento I don’t believe by looking to its code that is caused by the library, and instead I believe that I have hit some weird combination of factors that got Mnesia core confused on it’s start. You can check my post here.

• Also, Mnesia is not full ACID as claimed in the Erlang docs, but they say it his if you have enough replication(really ), see links:

  • Unexpected behaviour with mnesia - not actually durable?)
  • http://erlang.2086793.n4.nabble.com/mnesia-sync-transactions-not-fsynced-td4673313.html
  • MnesiaKV - new KV store ontop of RocksDB + ETS
    • Redirecting…
    • Redirecting…

• The good and bad in Mnesia from my research can be found here.

Thanks for the tips, especially the bit about starting tables

Regarding the thread you posted, and actually reading your thread in-depth really enriched my mnesia research

I’m using mnesia right now for saving/loading GenServer state for multiplayer gamestate, and at the moment we don’t need the persistence to be so reliable, but I have an eye towards a RocksDB-like or somesuch solution for persistence reliability should the need arise

@njwest we used mnesia for gaming too (its great for storing persistent state, as the lookup speed is ets (5m+ reads per second, from multiple schedulers) and the write is fairly fast 800k~ish, for a small server.

The way you do it is pretty much the way we did it (dont use mnesia anymore). Even if mnesia is not started as an extra_application, that :mnesia.stop ensures nothing weird happens.

Do you mind to say why you moved away from Mnesia?

Our usecase was for a durable fast KV store, think Cloudflare KV. That can be grepped. And ideally the objects map 1:1 with erlang terms (so we dont need to write SQL or another query language / deal with type inconsistencies).

What we thought we wanted was Mnesia. But it gave us a very complex multi use database that did not quite do everything as we wanted, lost data and was randomly slow under load.

What we really wanted was :ets that persisted to disk.

So we made :ets that persists to disk every write (not via tab2file that does not work), abate not 100% happy with the write performance of the older rocksdb version its using (newer rocksdb has concurrent unordered writes, and the workload is 100% write, since reads hit the ETS tables). Also eventually it needs to be distributed with some kind of simple routing algo (think etcd without strong consistency).

The project is called mnesia_kv, maybe the ideal version of it wont use rocksdb but just a simple WAL that gets aggregated somehow at thresholds (and is optimized for 100% write workload) but rocksdb was the quick and reliable choice at the time.

NOTE: we looked at patching Mnesia but the rabbit hole went too deep, it would take too many hours. Thats for a larger company like Facebook who has the resources to go down that route.

Why not just writing to a log file in the disk?

Since the first time I read the Kafka Design docs I have been fascinated with the power of directly writing to the disk, and now I have found another interesting reading:

I always have wanted to have a log of everything that occurs in my application so that I can recover in case of disaster, by replaying it, more or less like Event Sourcing but without all its overhead and complexity.

Now with this Mnesia issues, I also had though in just wrapping :ets with direct disk persistence or write another backend for Mnesia, but like you I think Mnesia is a rabbit hole, thus I think I will just try to write a simple disk log that can cope with high throughput, and then use it for wrapping :ets or has a backup of Mnesia to be used when a netsplit occurs or I lost all the data as happen to me already.

This repo may be a good starting point:

https://github.com/pbudzik/eventlog

But we already have another one in Erlang that is distributed:

https://github.com/erleans/vonnegut

Eventlog seems attractive to me, because is in Elixir, thus I can easily fork it and make it work as I intend, but I may try vonnegut and adopt it if doesn’t suffer of the same issues I am seeing now with loosing data due to the way it uses cache to write to the disk.

While reading this blog post I found that Mnesia is using selective receive blocks:

• otp/lib/mnesia/src/mnesia_checkpoint.erl at e0e9d8bad472fa94f4919e6db798e45913dc9eec · erlang/otp · GitHub
• otp/lib/mnesia/src/mnesia_locker.erl at e0e9d8bad472fa94f4919e6db798e45913dc9eec · erlang/otp · GitHub
• otp/lib/mnesia/src/mnesia_locker.erl at e0e9d8bad472fa94f4919e6db798e45913dc9eec · erlang/otp · GitHub

As the blog post mentions the selective receives can be a cause of slow down under load:

Conclusion

Selective receive is an interesting functionality that comes built into Erlang/Elixir. As with every other tool it has both strengths and weaknesses. Selective receive provides some advantages when working with relatively small message boxes (namely prioritised message processing), however, using selective receive without being aware of the potential costs can put your overall application stability at risk.

So, in your case maybe Mnesia was slowing down due to one of this cases.

Its not that simple, because I want the final data to be stored with periodic snapshots/log flushes, otherwise the larger the DB gets the significantly longer it will take to rebuild it from the first log entry. Like say 128MB WAL, once 128MB is reached, flush it into sst/another similar storage medium. If app crashes anywhere along the line, it only needs to parse max 128MB~ of journal and read the rest from the main ondisk table. If everything is a log, indeed its very simple, but the dataset will always grow. Say I do 1b inserts then 1b-1 deletes. That will be a size 2b-1 log, or a size 1 sst.

A netsplit is not that dangerous, if your just using a KV store, take the newest record and thats it. Or deepmerge the newer ontop the older one.

It could be, also it could be internally how the log persistance works.

I never really understood Kafka (and similar) and couldn’t find a good usecase for it.

This really depends on your use case and shape of the data.

To me sounds like a very dangerous shoot on your own foot gun…

It’s excellent for when you want to do Event Sourcing or want to have a decoupled MicroServices architecture. It requires another mindset and programming approach to your application.

The advantages of using a Kafka as a data lake that anyone in the organization can consume or produce data into is very powerful and opens endless opportunities for teams to progress without depending/waiting on each others.

Normally you will use Kafka to add data as facts after you have processed it, so that others can do whatever they want with it, like analytics/reports/audits, or just to trigger further actions. Another use is to serve as a backup for your application be able to recover from, but you need to have your application coded to support replay without performing the side effects, like sending emails or doing a wire transfer.

Eventual consistency. If you need CP (of CAP) then of course it is dangerous, but IMO you cannot have a low latency distributed system (planetary scale cringe) with CP, you must be CA.

What does this even mean?

As a datalake Kafka makes sense.