nathanl
Wanted: Your OTP architecture diagram
I’ve been reading Elixir in Action to get a better grasp of OTP. Part of the book works through building up an OTP app, and shows a diagram, which I’ve recreated here in ASCII using Monodraw:
┌────────────────────┐
│ Todo.Supervisor │
│ (rest_for_one) │
└────────────────────┘
│
┌────────────────┴───────────────┐
┌─────────▼───────────┐ ┌────────────▼─────────────┐
│Todo.ProcessRegistry │ │ Todo.SystemSupervisor │
└─────────────────────┘ │ (one_for_one) │
└──────────────────────────┘
│
┌───────────────────────────┼──────────────────────┐
┌───────────▼──────────┐ ┌─────────────▼────────────┐ ┌──────▼──────┐
│ Todo.PoolSupervisor │ │ Todo.ServerSupervisor │ │ Todo.Cache │
│ (one_for_one) │ │ (DynamicSupervisor) │ │ │
└──────────────────────┘ │ │ └─────────────┘
│ └──────────────────────────┘
│ │
│ │
┌──────────▼─────────┐ ┌───────▼───────┐
│Todo.DatabaseWorker ├─┐ │ Todo.Server ├┐
└─┬──────────────────┘ ├┐ └┬──────────────┘├┐
└┬───────────────────┘│ └┬──────────────┘│
└────────────────────┘ └───────────────┘
This diagram and the description of the app were very illuminating for me. @sasajuric tells why we spawn a fixed number of Todo.DatabaseWorker but a dynamic number of Todo.Server, why we use one_for_one on Todo.SystemSupervisor but rest_for_one on Todo.Supervisor, etc. (I would explain here if @sasajuric doesn’t mind, but don’t think it would be polite otherwise. Buy his book!
)
My request to you all: If you have a production OTP app and don’t mind, please share a diagram like this one of your application (or some interesting part of it) and explain the architecture choices in general terms. Maybe start with :observer.start(), then make an ASCII graph (eg with Monodraw or ASCIIFlow) and edit for clarity and/or discretion. I’m looking for things like:
- Which processes talk to each other and how do they find each other?
- Why did you choose multiple processes of type X but only one of type Y?
- Why did you choose to put A and B under different supervisors?
- Why did you choose the restart strategies you did?
I think this kind of high-level discussion would be very educational, without getting into all the details of the code.
Please share!
Most Liked
nathanl
Crickets, eh?
Alright, I’ll post one. Here’s part of a supervision tree from an application I’ve been looking at, built by @alexgaribay.
┌────────────────────────┐
│ Streams RootSupervisor │
│ (one_for_all) │
└────────────────────────┘
│
┌───────┴─────────────────────┐
▼ ▼
┌──────────────────────┐ ┌───────────────────────┐
│ Stream Info Fetcher │ │ Streams Supervisor │
│ │ │(dynamic, one_for_one) │
└──────────────────────┘ └───────────────────────┘
│
┌──────────────────────┴───────────────────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│Stream Supervisor │ │Stream Supervisor │
│ (one_for_all) │ │ (one_for_all) │
└──────────────────┘ └──────────────────┘
│ │
┌──────────┴──────────┐ ┌──────────┴──────────┐
│ │ │ │
▼ GenStage Pipeline ▼ ▼ GenStage Pipeline ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Stream Fetcher │ │ Stream Processor │ │ Stream Fetcher │ │ Stream Processor │
└──────────────────┘ └──────────────────┘ └──────────────────┘ └──────────────────┘
There are multiple incoming streams of data that need to be processed. We don’t know exactly what those streams are when the application boots.
StreamsRootSupervisor starts one Stream Info Fetcher, which makes a network request and gets back a list of the streams. Stream Info Fetcher tells Streams Supervisor to start one Stream Supervisor for each stream we need to process. This is why Streams Supervisor is “dynamic” - we start its children as we discover that we need them.
Each Stream Supervisor is in charge of making sure one stream of data is processed. To that end, it starts a Stream Fetcher to pull down data for that stream and a Stream Processor to process it, with the two connected in a GenStage pipeline.
Those are all the parts. Regarding supervision strategies:
- At the top,
Streams RootSupervisorusesone_for_allbecause its two children depend on one another; if one crashes, they both need restarting. Streams Supervisorusesone_for_onebecause each stream is processed independently; if one crashes, the others should carry on.Stream Supervisorusesone_for_allbecause aStream Fetcherand itsStream Processerare useless without each other; if one crashes, they both need restarting.
Note that once Stream Info Fetcher has done its work at boot time, it’s no longer needed and terminates. This doesn’t trigger StreamsRootSupervisor to restart all its children unless Stream Info Fetcher actually crashes; because Stream Info Fetcher is :transient; it’s expected to terminate normally and not be restarted.
bibekp
Great idea for a thread! Yes, Elixir in Action rocks!
I wrote this a couple of years ago. Here’s my diagram with fun colors!
Parallel hangman game play works by spinning up pairs of player workers and game servers (which talk to each other). The rest of the process tree is mostly administrative support. There’s more diagrams below on the github.. I didn’t get around to diagramming the GenStage dictionary ingestion pipelines.
axelson
Thanks for creating this topic! I think it’s a discussion that will be very enlightening for all involved.
I know that it is not quite what you want but I really like ElixirConf 2016 - Selling Food With Elixir by Chris Bell which has a great description of how they split up setup their OTP architecture (and that link goes to that portion of the talk). I especially find it cool how their design mirrors their business domain.
Popular in Discussions
Other popular topics
Categories:
Sub Categories:
Forums
Popular Tags
- #ecto
- #liveview
- #troubleshooting
- #learning-elixir
- #deployment
- #library
- #erlang
- #testing
- #genserver
- #mix
- #absinthe
- #remote-other
- #otp
- #plug
- #how-to-question
- #macros
- #postgres
- #channels
- #elixirconf
- #exunit
- #discussion
- #code-sync
- #javascript
- #podcasts
- #onsite
- #dialyzer
- #docker
- #authentication
- #umbrella
- #full-time-contract
- #podcasts-by-brainlid
- #ecto-query
- #elixir-ls
- #phoenix_html
- #iex
- #blog-post
- #graphql
- #genstage
- #ai
- #websockets
- #supervisor
- #advent-of-code
- #elixirconf-us
- #distillery
- #processes
- #forms
- #api
- #metaprogramming
- #security
- #performance










