Harrisonl

Distributed elixir or Message broker for microservice communication?

Trying to decide between using a message broker or queue services (e.g. RabbitMQ) for passing messags between the microservices OR using distributed elixir and passing messages via processes/RPC. Keen to hear the pro’s and cons of one over the other.

12 comments

#distributed-systems

2 7022 12

2019-03-12 13:52:49 UTC

Most Liked

tristan

Rebar3 Core Team

My suggestion is always, only use distributed erlang if it is for a control plane.

If you are talking between services there is a lot to say about being able to reuse existing infrastructure for **** like load balancing (like http and recently grpc, which I’d suggest personally, has more support).

Distributed Erlang gives you a single non-multiplexed connection between each node, so workloads that you’d expect to work mostly fine if you were doing http, like with large payloads, will fall apart.

There is work on improving distributed Erlang (chunking large messages was recently added and I guess will be in OTP-22) and there is partisan (http://partisan.cloud/) for a high performance distributed Erlang. But for what it sounds like you are building you will likely have success using http or grpc.

Post #5

NobbZ

As in distributed erlang each node has to connect to each other node, you’ll end up with connections you do not need.

Therefore, I’d go for REST, a message broker or any other ISC that does not require a fully meshed network.

Post #2

Harrisonl

@Ciboulette the way I see it, you would want to use distributed elixir when you are building more of a single distributed system where alot of the components will be interacting with each other. Essentially when you connect all the nodes up, it becomes one big system/application which has acess to all other parts of the system (correct me if I am wrong here though).

Whereas on the other hand, if you are building a set of small, de-coupled micro services that don’t really need to be apart of a distributed system, then using something like a message broker might be the best bet.

I think personally I would typically use the distributed section if I had a single application that I need to be HA, (e.g. deployed over multiple servers) but also wanted it to be able to communicate with all other running versions of it’s self (for things like interval jobs for example) and then use a message broker to communicate with other services/applications in the wider network. So basically, a combination of both if that makes sense. However there are definitely still cases where it might make sense to have two seperate applications connected via distributed erlang/elixir but I think it’s more of a problem specific question in that case

Post #10

Last Post!

easco

As I understand it, by default, Erlang will create a fully connected mesh of nodes. What that means is that if you have n machines in the mesh, each node, by default, will connect to the other n - 1 nodes. With that fully connected mesh, each connection requires a little bit of overhead and at some point that overhead becomes overwhelming.

The key words in the above, however, are by default. You can take greater control over the ways that the machines are connected - you don’t have to use a fully connected mesh. One configuration I’ve read about is a hash-ring where the computers are connected so that they know one or two other nodes, and form a ring. There could very well be more than 50 nodes in the ring, but it’s not possible to directly send a message from each node, to every other node.

So your ability to create large networks of Erlang nodes is affected by the topology of the connections.

Post #13