Mnesia cluster - malicious node

I am building a cluster with nodes on elixir and I am considering using mnesia as a distributed database. The plan is the nodes to send messages among each other with RabbitMQ (AMQP library). In the future more nodes will join the cluster, but the biggest concern is that they are different parties and do not trust each other.

My question is: if a new node joins, runs iex and just decides to drop a shared table or delete records, does anything stop it from doing that?

When using sync_transaction for CRUD operations I know that all the other nodes should approve the transaction in order to be completed. But that is not the case when a node can freely decide what to do by running commands in the terminal.

If there is no way of preventing that, are there any alternatives of mnesia for this scenario?

If notes are connected to each other using distributed Erlang (EPMD or similar), then any node can wreak total havoc on the cluster, because nodes are not shielded from eachother. Distributed Elixir/Erlang is meant to be used in situations where all nodes can be trusted.

Instead, you will need some other way to manage the communication between the nodes, which does not allow full access to remote nodes. You will probably require some kind of consensus system that is Byzantine fault-tolerant.

2 Likes

Distributed erlang does not handle untrusted parties, at all, not even a little. Communicating via RabbitMQ is fine, but use a transport that isn’t distributed erlang.

I’m a little confused about why you want distributed mnesia or erlang between nodes if you already have a communication method (RabbitMQ).

1 Like

The idea is that the nodes share a database which is a whitelisting of addresses. Moreover, if a node fails and goes down, it should be able to retrieve any added addresses in the whitelist while it was down, even the node that added them also goes down at some point. Having at least 3 nodes (1 of which is up all the time) achieves that.

I am not sure if RabbitMQ can do that?

It cannot, but mnesia won’t give you a workable security model for that either. You’ll want to find some other kind of common store. Mnesia and distributed erlang were originally built for co-located machines on the same physical rack to enable failover. Those mechanisms to some extent translate to a cloud environment, but they are not general purpose communication methods. If you can’t trust the nodes, you can’t use mnesia or distributed erlang between the nodes.

2 Likes