Architecting Systems with Elixir

There’s no need to do either actually since each genserver pool is supervised and could be considered a service. BEAM has its own communication fabric, so gRPC is also unnecessary unless you are planning to do polyglot services.

I’d just start with a regular phoenix project and add modules / genservers as required.

Ohh buddy, you’re gunna stir the hornet’s nest with this one!

My day job has a similar setup as you. We started with an umbrella app, but then just merged everything into a monolithic Elixir app with multiple top level namespaces… and I think it’s MUCH more simple.

We also use Kubernetes… but it’s more about individually scalable deployments, not code separation. We have dozens of Kubernetes deployments, most of them hooked up to HPA, but they all use a shared “Elixir app” build image.

Also, the concept of “RPC” in a dynamically typed language is interesting to me… We have “RPC” which is really just API calls over BSON+NATS (as opposed to JSON+HTTP). But is that really RPC? Isn’t RPC all about static type analysis, i.e. making remote calls look like local calls, including static type checking?

My suggestion would be to roll with a single repository, no umbrella, specially if you are a small team.

For example, if your Broadway pipeline may need to use some database resources, and if that’s on a separate application, now you need either a separate RPC service or an additional application to encapsulate shared logic.

The benefit of the Erlang VM is exactly to juggle many things at once (Go as well?), umbrella or not.

I would go with umbrella and/or separate services if you have a larger team and you would naturally organize around some segments or if certain parts of the application have distinct scalability needs.

PS: using gRPC with Elixir is also completely fine.

I’d add to the already given suggestions, that elixir is usually rather nice to refactor. So this is really less of an “either-or” question as one might expect. In other languages I’ve seen people opt to start out with the most complexity from day 1, because the expectation is that things become a big ball of mud and you cannot change things later. To a degree this might happen with elixir as well, but the functional nature of elixir will allow you to change things quite a bit easier, so you really can start with the MVP (even in architecture) and change things only when there’s demand for it.

Do you mean using a single mono-repo, with multiple apps/projects in there?

Ah, good catch. I meant a single project/app.

Does your folder structure look like this?
/foo_app/lib/foo_app
/foo_app/lib/foo_app_web
/foo_app/test
/bar_app/lib/bar_app
/bar_app/lib/bar_app_web
/bar_app/test

Do those apps share the same Beam VM?
How many application.ex files do you have? If one it’s in the root folder?
How are third-party dependencies managed? Each app has its mix.exs file?
How do you manage the dependencies between the apps? And how do they communicate?

Considering this is an alternative to umbrella apps for many cases, for which cases will we need umbrella apps? In other words, what will umbrella apps solve that top level namespaces can’t solve?

I would use Phoenix exclusively for the web logic and then everything else that needs business logic not tied up with serving the request will go into a poncho project.

A poncho project is just a simple elixir project in a parent folder that is then required as a dependency on the Phoenix web app.

See an example form following @PragDave course in this repo:

.
├── dictionary
│ ├── _build
│ ├── assets
│ ├── lib
│ └── test
├── hangman
│ ├── _build
│ ├── deps
│ ├── lib
│ └── test
├── hangman_live
│ ├── _build
│ ├── assets
│ ├── config
│ ├── deps
│ ├── lib
│ ├── priv
│ ├── rel
│ └── test
└── text_client
├── _build
├── lib
└── test

The Phoenix App is in hangman_live while the core logic for the game is in hangman. The text_client is another way of playing the game, thus it also uses hangman which is agnostic of how the game is being played, in a browser or in a terminal, doesn’t matter. The dictionary was extracted from hangman because its generic and can be reused in other apps.

Example of requiring hangman as a dependency from hangman_live can be seen here:

{:hangman, path: "../hangman"},

This is what I like to cal a clean architecture that gives you the benefits of mircroservices in a mono-repo, but without adding any great deal of complexity and keeping things decoupled and reusable. If later you decide that you really need a microservices architecture then you are almost there.

No, our dir structure is vanilla Phoenix app, but with top level namespaces in lib…

/our_app/lib/our_app/application.ex
/our_app/lib/our_app/*
/our_app/lib/our_app_web/*

/our_app/lib/foo/*
/our_app/lib/bar/*

I.e. Foo and Bar are top level namespaces, and we have only one BEAM application which is :our_app.

If it wasn’t clear, I meant we reverted from umbrella apps to just normal Elixir/Phoenix apps, i.e. whatever is spit out by mix new or mix phx.new. And we don’t mind having top level namespaces. So instead of OurApp.Datastores, we simply have Datastores.

We conditionally start processes/supervisors based on environment variables, so something like this in application.ex…

children = if System.get_env("START_WORKER") do
  [OurApp.Worker | children]
else
  children
end

I don’t see much benefit to umbrella apps.