nathanl

nathanl

Wanted: Your OTP architecture diagram

I’ve been reading Elixir in Action to get a better grasp of OTP. Part of the book works through building up an OTP app, and shows a diagram, which I’ve recreated here in ASCII using Monodraw:

                ┌────────────────────┐
                │  Todo.Supervisor   │
                │   (rest_for_one)   │
                └────────────────────┘
                           │
          ┌────────────────┴───────────────┐
┌─────────▼───────────┐       ┌────────────▼─────────────┐
│Todo.ProcessRegistry │       │  Todo.SystemSupervisor   │
└─────────────────────┘       │      (one_for_one)       │
                              └──────────────────────────┘
                                            │
                ┌───────────────────────────┼──────────────────────┐
    ┌───────────▼──────────┐  ┌─────────────▼────────────┐  ┌──────▼──────┐
    │ Todo.PoolSupervisor  │  │  Todo.ServerSupervisor   │  │ Todo.Cache  │
    │    (one_for_one)     │  │   (DynamicSupervisor)    │  │             │
    └──────────────────────┘  │                          │  └─────────────┘
                │             └──────────────────────────┘
                │                           │
                │                           │
     ┌──────────▼─────────┐         ┌───────▼───────┐
     │Todo.DatabaseWorker ├─┐       │  Todo.Server  ├┐
     └─┬──────────────────┘ ├┐      └┬──────────────┘├┐
       └┬───────────────────┘│       └┬──────────────┘│
        └────────────────────┘        └───────────────┘

This diagram and the description of the app were very illuminating for me. @sasajuric tells why we spawn a fixed number of Todo.DatabaseWorker but a dynamic number of Todo.Server, why we use one_for_one on Todo.SystemSupervisor but rest_for_one on Todo.Supervisor, etc. (I would explain here if @sasajuric doesn’t mind, but don’t think it would be polite otherwise. Buy his book! :smile: )

My request to you all: If you have a production OTP app and don’t mind, please share a diagram like this one of your application (or some interesting part of it) and explain the architecture choices in general terms. Maybe start with :observer.start(), then make an ASCII graph (eg with Monodraw or ASCIIFlow) and edit for clarity and/or discretion. I’m looking for things like:

  • Which processes talk to each other and how do they find each other?
  • Why did you choose multiple processes of type X but only one of type Y?
  • Why did you choose to put A and B under different supervisors?
  • Why did you choose the restart strategies you did?

I think this kind of high-level discussion would be very educational, without getting into all the details of the code.

Please share!

Most Liked

nathanl

nathanl

Crickets, eh? :laughing: Alright, I’ll post one. Here’s part of a supervision tree from an application I’ve been looking at, built by @alexgaribay.

         ┌────────────────────────┐
         │ Streams RootSupervisor │
         │     (one_for_all)      │
         └────────────────────────┘
                      │
              ┌───────┴─────────────────────┐
              ▼                             ▼
  ┌──────────────────────┐      ┌───────────────────────┐
  │ Stream Info Fetcher  │      │  Streams Supervisor   │
  │                      │      │(dynamic, one_for_one) │
  └──────────────────────┘      └───────────────────────┘
                                            │
                     ┌──────────────────────┴───────────────────────┐
                     ▼                                              ▼
           ┌──────────────────┐                           ┌──────────────────┐
           │Stream Supervisor │                           │Stream Supervisor │
           │  (one_for_all)   │                           │  (one_for_all)   │
           └──────────────────┘                           └──────────────────┘
                     │                                              │
          ┌──────────┴──────────┐                        ┌──────────┴──────────┐
          │                     │                        │                     │
          ▼  GenStage Pipeline  ▼                        ▼  GenStage Pipeline  ▼
┌──────────────────┐  ┌──────────────────┐     ┌──────────────────┐  ┌──────────────────┐
│  Stream Fetcher  │  │ Stream Processor │     │  Stream Fetcher  │  │ Stream Processor │
└──────────────────┘  └──────────────────┘     └──────────────────┘  └──────────────────┘

There are multiple incoming streams of data that need to be processed. We don’t know exactly what those streams are when the application boots.

StreamsRootSupervisor starts one Stream Info Fetcher, which makes a network request and gets back a list of the streams. Stream Info Fetcher tells Streams Supervisor to start one Stream Supervisor for each stream we need to process. This is why Streams Supervisor is “dynamic” - we start its children as we discover that we need them.

Each Stream Supervisor is in charge of making sure one stream of data is processed. To that end, it starts a Stream Fetcher to pull down data for that stream and a Stream Processor to process it, with the two connected in a GenStage pipeline.

Those are all the parts. Regarding supervision strategies:

  • At the top, Streams RootSupervisor uses one_for_all because its two children depend on one another; if one crashes, they both need restarting.
  • Streams Supervisor uses one_for_one because each stream is processed independently; if one crashes, the others should carry on.
  • Stream Supervisor uses one_for_all because a Stream Fetcher and its Stream Processer are useless without each other; if one crashes, they both need restarting.

Note that once Stream Info Fetcher has done its work at boot time, it’s no longer needed and terminates. This doesn’t trigger StreamsRootSupervisor to restart all its children unless Stream Info Fetcher actually crashes; because Stream Info Fetcher is :transient; it’s expected to terminate normally and not be restarted.

bibekp

bibekp

Great idea for a thread! Yes, Elixir in Action rocks!

I wrote this a couple of years ago. Here’s my diagram with fun colors!

Parallel hangman game play works by spinning up pairs of player workers and game servers (which talk to each other). The rest of the process tree is mostly administrative support. There’s more diagrams below on the github.. I didn’t get around to diagramming the GenStage dictionary ingestion pipelines.

https://github.com/brpandey/hangman

axelson

axelson

Scenic Core Team

Thanks for creating this topic! I think it’s a discussion that will be very enlightening for all involved.

I know that it is not quite what you want but I really like ElixirConf 2016 - Selling Food With Elixir by Chris Bell which has a great description of how they split up setup their OTP architecture (and that link goes to that portion of the talk). I especially find it cool how their design mirrors their business domain.

Where Next?

Popular in Discussions Top

sashaafm
Piggy backing a bit on @dvcrn topic BEAM optimization for functions with static return type?, I’ve been trying to understand in a deeper ...
New
mbenatti
Following https://github.com/tbrand/which_is_the_fastest |> https://raw.githubusercontent.com/tbrand/which_is_the_fastest/master/imgs...
New
AlexMcConnell
The reason that Rails is as popular as it is is because it’s very easy for relatively inexperienced developers to get a lot of work done....
588 19652 166
New
ben-pr-p
In general I’ve been sticking to this community style guide GitHub - christopheradams/elixir_style_guide: A community driven style guide ...
New
wmnnd
The Go vs Elixir thread got me thinking: Would it be too hard to implement a simple mechanism for creating Go-style static app binaries f...
New
tmbb
This is a post to discuss the new Phoenix LiveView functionality. From Chris’s talk, it appears that they generate all HTML on the serve...
342 18243 126
New
gausby
I asked this very same question on twitter and got some interesting feedback, but I thought it would be a good question to ask here as we...
1207 39467 209
New
opsb
We’re considering our architecture from a viewpoint of scaling our traffic heavily over the next 6 months. Our current deployment is runn...
New
RudManusachi
What configs will make sense to put to runtime.exs? – A bit of how I configure apps: I have generic configs in config/config.exs, dev...
New
mmmrrr
Just saw that dhh announced https://hotwire.dev/ Is it just me or is this essentially live view? :smiley: Although I like the “iFrame-e...
New

Other popular topics Top

JeremM34
Hello, how can I check the Phoenix version ? Thanks !
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod – where is this set? Thanks.
New
Darmani72
If I have a post route which an argument: post /my_post_route/:my_param1, MyController.my_post_handler How would get the post params ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
AstonJ
Seen any cool LiveView demos, sample apps or examples? Please post them here! :003:
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 43757 214
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
dblack
I’ve got an issue with an app and I’ve no idea of how to troubleshoot it. I’m hoping someone here might have seen something similar. I p...
New
sergio
Kind of like when jquery came out, it was super necessary. Existing drag and drop libraries have a bunch of baggage to support old browse...
New

Latest on Elixir Forum

Elixir Forum

We're in Beta

About us Mission Statement