Elixir nodes synchronization and load balancing

Hello there.

I have a question regarding how to handle load balancing and zero downtime in case of server crashes for my specific use case.

First I will explain what my system architecture looks like as show in the image bellow:

one_server2698×559 8.44 KB

So, my Server A consists in a bunch of elixir nodes, one node is the Phoenix node which my clients can make requests.

The Fetcher node is in charge of connecting to third-party endpoints (via rest and websocket) and fetch new data from them. This new data is processed and will be sent to the Notification node.

The Notification node is in charge of sending push notifications to the clients. First the client sends its app token to the Phoenix node and a list of type of notifications he is interested in receiving, next the Phoenix node sends this token and list to the Notification node so it knows to who send the new data received from the Fetcher node when it arrives.

Server A is inside one server hardware right now (due to budget constraints), but this is not good since if this hardware goes down for some reason the whole system goes down too. So I was thinking on how could I duplicate this structure inside another server so if one goes down the other is used, something like this:

two_servers806×809 13.6 KB

Now I have Server A and Server B, if one goes down the other will still (hopefully) be working. Also, both Fetcher Node A and Fetcher Node B would be connected to the same third-party endpoints, so both will receive the same external data.

The issue that I can see with this approach is in the Notification nodes.

The first one is that both of them will be sending notifications to the clients, so the clients will receive duplicated notifications. To fix that I need some way to elect one of the two Notification nodes to be the master and allow only this node to send notifications. Also I need to know if the master node goes down so the slave one can start sending the notifications until the master node goes up again.

The second issue is about synchronization of client tokens and list of desired notifications. Since the load balancer will distribute the client requests between both servers, I need a way to keep data about the client tokens, desired notifications synced between both Notification nodes.

Do you guys have any suggestion of what is the best way to handle these cases in Elixir?

Thanks for the help.

Update:

I just found a library called ra from rabbitmq which implements the Raft concensus algorithm.

Seems like a great possible solution for my case. There is a great talk about it here btw: link