fireproofsocks

fireproofsocks

I’m working to increase my understanding of how to build distributed systems as a way to leverage concurrency. I understand this is a broad topic, but I’m wondering if anyone has some examples of how to structure “work” across multiple nodes or multiple apps.

I admit I am not even sure I follow how things happen when you deploy a single app across multiple nodes (e.g. 1 app deployed to 2 EC2 instances)… that means there are 2 instances of the Erlang VM running. How is work delegated? E.g. if it’s a Phoenix app, does a multiple-server deployment assume that a load balancer is figuring out the “division of labor” and routing requests to different nodes? And does this assume that each node more or less has no awareness of the other nodes?

Or how about when you want to process a stream of inputs (e.g. using Task.async_stream/3)… do you have any way of routing the processing to different nodes? For this to work, wouldn’t each node need to know about the other nodes? Locally, this happens by specifying --sname and --remsh options when starting the app, e.g. something like

iex --sname app1@localhost -S mix
iex --sname app2 --remsh app1@localhost

What are good questions to ask when deciding how to break up work across multiple nodes/apps?

Showing Posts 1 to 3

DidactMacros

DidactMacros

Hi, I’ve recently read about distribution from the Elixir in Action book, 3rd edition.

The below advice is from my understanding of that book.

You could delegate (compartmentalise specialised work) using process groups.

Here is the documentation for the :pg module.

You would thus have a set of nodes that the other nodes can know are is responsible for doing particular jobs.

You would also have processes that serve particular purposes registered at initialisation, they would have their own PIDs that would include the host addresses identifiable by remote nodes anyway, so you could forward requests to those processes even though they are on remote nodes using very similar messaging sending procedures.

You could forward requests to a request resolving process that would identify which process is in charge of what service, or even request the spawn of a process by a pool manager in order to service a given request.

You would want to use calls to do this, to receive confirmation that the request was successfully served. Here is the documentation for remote procedure calls.

This probably isn’t a whole lot of info, just pointing you in the right direction, I’d definitely consider giving chapters 10 - 12 from Elixir in Action a read.

viniciusmuller

viniciusmuller

Hey, have you checked the “Introduction to Mix” guide in the official documentation?
Although the name can be a bit misleading, this is the description of the guide:

In this guide, we will build a complete Elixir application, with its own supervision tree, configuration, tests, and more.

The application works as a distributed key-value store. We are going to organize key-value pairs into buckets and distribute those buckets across multiple nodes. We will also build a simple client that allows us to connect to any of those nodes and send requests

I think it might be helpful, it covers how nodes connect to each other using the configurations and shows some example of running code in a remote node and distributing work across nodes via sharding.

Overall the erlang runtime does no magic to help you and you still need to build your own mechanisms of distribution and load balancing in your application, but it provides all the building blocks you’ll need to do so.

apsori

apsori

I really need to get a copy of this book :slight_smile:

— All posts loaded —

Where Next? Top

Trending in Discussions Top

AstonJ
As the title says, please share what you’ve been up to with Elixir. Whether that’s been learning it, looking into it, making stuff with i...
2977 92995 915
New
caslu
I want to open this thread for you all to discuss and help those who really like Ash but are still hesitant to use it in a real project. ...
New
mudasobwa
I am happy to introduce the very α version of the new programming language compiled to BEAM. Welcome Cure. It has literally three kille...
New
GES233
I’m posting this in response to Jose’s recent tweet (Cr. link) : People are sleeping on Elixir for a coding harness: Hot-code swappi...
New
_mfierro
Hello, I wrote Stop My Hand, a Scattergories-like web application using Phoenix/LiveView as my learning project for Elixir (after readin...
New
marciol
It would be helpful to have a list of companies worldwide that hire engineers without prior experience in Elixir. Often, it can be quite ...
New
durvia
Anyone running long-lived stateful processes on BEAM? We’re building an AI agent runtime and would love to compare notes. We’re a small ...
New

Other Trending Topics Top

garrison
Hobbes is a low-level distributed database for the Elixir programming language. Hobbes provides a simple, safe, and scalable storage lay...
New
marciok
Hi there! We created Gust: A task orchestrator inspired by Airflow. For those who have never heard about Aiflow, it’s a Python-based wor...
New
jimsynz
Beam Bots (or just BB for short) is a framework for building fault-tolerant robotics applications in Elixir using familiar OTP patterns. ...
New
Dmk
Xamal is a deployment tool for Elixir apps that deploys native releases to bare metal servers over SSH. It’s a port of GitHub - basecamp/...
New
Damirados
Hello everyone. After busy few months I am happy to announce v0.1.0 of Emerge & Solve. They are GUI (Emerge) and State management (S...
New
wintermeyer
There are three potential reasons for members of this forum to have a look at https://vutuv.de You are tired or annoyed of LinkedIn. Yo...
New

We're in Beta

About us Mission Statement

Options

Thread Display Mode




Thread Preview

Skip Thread Previews