Packaging up separate apps in a single distillery deploy like Dave's Hangman app

Questions / Help
1

I finished up Dave’s Elixir for Programmers course (really enjoyed it, review coming soon) but it did leave me with one big question coming out of it.

Dave emphasizes the non-umbrella, multi-application approach. I like everything about the approach from a separation standpoint. My biggest question was about how to deploy it as a single unit so that it could be deployed on say AppEngine, Kubernetes, Heroku, Gigalixir, etc.

Aside from Heroku, most deployments revolve around Distillery. What would be the proper way to package up those apps in a release?

Here’s the public code from the end of the course:

Essentially you have an app with:

  • Dictionary (word list process)
  • Hangman (game logic that uses the dictionary)

And then 3 different interfaces to the game:

  • Text Client
  • Gallows (web client)
  • Socket Gallows (channels)

The two big questions I have with this approach vs umbrellas are around deployments on a single BEAM and dependencies.

In the Phoenix docs discussing Umbrellas there is a Don’t Drink the Kool-Aid section that mentions the following:

If you find yourself in a position where you want to use different configurations to different applications in the same umbrella or to use different dependency versions, then it is likely your codebase have grown beyond what umbrellas can provide.

The good news is that breaking an umbrella apart is quite straightforward, as you simply need to move applications outside of the umbrella project’s apps/ directory. In the worst case scenario, you can discard the umbrella project and all related configuration (build_path, config_path, deps_path and lockfile) and still leverage the “mono-repo” pattern by keeping all applications together in the same repository. Each application will have its own dependencies and configuration. Dependencies between those applications can still be explicitly listed by using the :path option (in contrast to :git).

From reading that it makes it sound as if:

  1. The Umbrella would have a shared dependency tree, so if a dependency in one app had a hard requirement on a specific version of a package all of the dependencies within the umbrella would be stuck with that.

  2. The non-umbrella version would be able to still live as a single mono-repo and presumably deployed as a unit…but have their own untangled dependency trees that could include different versions of the exact same package for each application’s own uses.

If that is actually the case, it definitely seems like the non-umbrella approach would win best architecture with a landslide as long as the deployment questions are easy enough to answer.

3 Likes
2

I very much prefer applications being dependencies and not using an umbrella myself. ^.^

2 Likes
3

Any special concerns to deploy in that setup?

4

Not that I’ve had. Any interdependencies I just configure module names via the configs (which makes testing a lot easier too).

5

I may be totally off-base but can’t you only have one version of each application on the BEAM? So if you use a mono-repo approach with multiple applications, all the versions still need to match-up anyway.

Also, Nerves calls this approach (perhaps unofficially) a “poncho project”

2 Likes
6

Has anyone deployed a project like this with distillery?

7

I have an umbrella with apps for different targets and so a release should consist of a subset of these apps. The approach I take there should work for this situation too:

Create a “master” app which depends on the apps you want included in the release. This gives you a single command to start/stop all the applications in one go.

1 Like
8

I’m curious if you ever came to any conclusions regarding this. I’ve been experimenting with a “poncho” style application and it seems to have the same issues as an umbrella app (shared dependencies, shared config). I’m having a hard time not seeing a poncho application as a less convenient umbrella application.

9

I can’t really say that I’ve come to any real conclusions. I believe it’s impossible to get away from the shared dependencies issue on the BEAM, but shared configuration is a difference between the poncho and umbrella approaches. So the benefit of the poncho approach is that you can have separate config for the higher-level applications in the poncho, while in an umbrella application you have one shared global configuration for all the applications in the umbrella.

1 Like
10

I actually just had a really good conversation over on Slack and they confirmed that non-shared dependencies are impossible on a single BEAM instance.

Good to know on the config side. Not sure how to make that work with distillery though. I’ve either had to have one big poncho config for all apps or import all the disparate app configs into the poncho which I believe smoosh’s them together like you would in an umbrella. Not sure what other options are available.

11

So i just explained how I setup large scale Elixir appliactions on the #nerves Slack channel. I’ve seen this come up a lot lately, so i think i’ll copy-pasta bits of my post from Slack here. Maybe it can help someone out.

begin copy paste mode

FWIW poncho and umbrella are pretty much the same thing, just poncho discourages sharing global application configuration and a root level mix.exs. Basically what Dave describes, before named before he described it. i personally think Elixir Umbrellas are somewhat broken or not super usable.

what i usually do [for a Nerves application] is

mkdir new_project 
cd new_project # git init here
mix new new_project_core # core business logic. This may itself be later broken up.

# the following is the app that gets "deployed". 
# You can think of this as the production wrapper. 
mix nerves.new new_project_firmware 

mix phx.new ui # user interface part of the application.

Then new_project_core has 0 dependencies on the other applications inside the “poncho”.

new_project_firmware depends on both new_project_core and new_project_ui
new_project_ui depends only on new_project_core and NOT new_project_fw
After about ~3 or so years, i’ve found this to be the most maintainable way to build a Nerves application, however it is sort of overkill for simple applications and requires a bit of forward thinking that you may not have time for in early stages of a project

You will need to duplicate configurations in some cases. core will only have configuration for it’s own runtime/test envs which may be completely different than ui, and fw. fw. fw will likely have phoenix config along with nerves config. ui will only have phoenix config.

I don’t just do this for Nerves, but any large scale Elixir application.

An Example

A small (huge) example might be useful: https://github.com/farmbot/farmbot_os/tree/next

there are a number of sub applications:
farmbot_core - Non networked stuff. Depends on Ecto.
farmbot_ext - things that are networked. AMQP, HTTP, etc.
farmbot_os - This is the Nerves app. Production config with OS specific stuff. (this is what would be deployed to say Gigalixir or Heroku if this were a web application instead of a Nerves application. Mostly glue code and supervision for the other sub systems/apps)

  • core only has configuration for its dependencies.
  • ext has configuration for core, (including it’s dependencies) and specific deps such as Tesla, AMQP, etc.
  • os has configuration for core, and ext. (including their dependencies) along with it’s specific deps such as nerves.

Also I should say: This may not be the best or even a good way of structuring an application. It’s just how I do it. As always your results may very.

4 Likes
12

Thanks for the great info. With the non-shared dependencies on a single beam being a non-starter I actually feel a lot more strongly about just using umbrella and contexts until it’s no longer tenable.

The main reason I brought up the question is that there always seems to be this dichotomy between Chris and Jose trying to balance capability and architectural ideals against developer simplicity and efficiency.

I usually firmly fall into camp “balance” over camp “purity” but if Dave’s approach solved the shared dependency issue it would have won me over. Without solving that issue, the approach is similar to going to micro services too early.

I’ve read through the beta of programming Phoenix 1.4 and the coverage of contexts is almost exactly what Dave advocates for, minus the multi node complexity before it’s needed.

Separating things out when the time comes is tremendously simpler in an Elixir app than an OOP equivalent, so I’m firmly back in camp “balance” because developer efficiency and time to market is always going to be more important until you have enough income and traffic to justify focusing on architectural purity (for web, not necessarily for nerves).

4 Likes
13

That’s a good point to make. It took three years of evolution, 5 Elixir minor versions, many Nerves revisions etc to arrive here for me personally. I don’t think there is one particular way of structuring a large scale application of any type or language.

What I like is that Elixir gives us the option and tools to do whatever we want with our project structure.

14

Do you use distillery at all? Not sure if that’s a think in the Nerves world.

15

Nerves does use Distillery. It works great.