How to create a distributed and loadbalanced system?

You might want to use libcluster with swarm, from the same author.

What exactly are you trying to do? The architecture will depend on what your ultimate goal

I came from a Service Based Architecture mindset, so maybe I am approaching this in the wrong way, but I was thinking on a system build by different Elixir apps that would provide different services.
Given a cluster of elixir nodes each app could be executed on any of those nodes.
When more resources are needed new nodes will be added to that cluster, manually or automatically, and tha apps running on those new nodes should be added to the existing ones and the load balanced between them to handle the increased traffic in the application.

That’s more or less what I was thinking. Seems to me that libcluster could help with the automagic discovery of new nodes, but do not know about the loadbalance part

You should also check Phoenix Pubsub, as standalone… It includes tools from now obsolete Firenest.

Also I am not sure it is a good idea to apply service based architecture concept to the BEAM, because it does not really work that way. Already one node is composed of multiple “micro services”, as any process could provide a service.

Have you thought about using a message broker instead of doing everything by yourself?
For this kind of distributed system I’d use RabbitMQ (which is written in erlang)

There’s something about the way that you posted this suggestion that bugged me. I have only been in prod for a short time, so I don’t really know, but I guess my gut feeling is you shouldn’t cluster everything and spawn up services dynamically inside your erlang cluster, just because you can. Maybe I’ve drunk the cool-aid, but one of the core philosophies behind BEAM systems is to give you the ability to reason about and manipulate failure domains. If you merge everything into a flat service mesh, you will lose that, and potentially lose control if by chance you wind up with a bunch of cpu heavy processes on the left side of the mesh and a bunch of io heavy processes on the right side of the mesh. Also, realistically to do what you want to do, you’ll need to organize your code into a single giant monorepo, which may not be what you want.

I think typically what you should do is group your services into “services which ought to come up and down together”, or “services which need security isolation” and use “non-erlang-distribution” methods to pass information between those services.

As I said maybe I am approaching this from the wrong point ( I have been looking for books regarding design best practices in Erlang but I have found no one yet) .

I was thinking that a node could be running a bunch of different “services” (apps) instances depending on the release type installed on it, that way I could add more app instances when necessary … the problems are:

• How to know which apps are running in the new node
• How to balance load to include the new apps instances running in the new node.

I am pretty new to Erlang/Elixir and I still trying to learn and understand the design best practices for this platform, so maybe I am totally wrong …

The failure domains concept you mention seems very interesting. Where could I read something more about those core design philosofies? I am really interested.

Regarding the big mono repo … well I am currently working on a multi-repo micro service solution and … it is a nightmare. In my experience so much repos add too much complexity without any reward …but that´s another story.

Thanks for your time and your answers …