Umbrella projects vs. stand alone apps as a dependency

Chat / Discussions
, , , ,
1

I’m going through Dave Thomas’s video course, “Elixir For Programmers”. In the course he talks about decoupling & maintaining that decoupling.

He recommends seperate apps over an umbrella project.

What he says makes a lot of sense, but I believe there must have been a lot of thought going into the umbrella project design.

I don’t have enough programming experience to comment & would like to hear the counter arguments.

(For more context, a “rant” quote from Dave:)

A Personal Rant About Kitchen Sink Web Apps
When you create a new Phoenix application, you end up with over 7,000 files spread over more than 1,000 directories—a total of about 45Mb.

Now most of these are not part of Phoenix itself—they are third party dependencies. In fact, over 90% of this is JavaScript.

But, ignoring all this, we’re still faced with the potential for another kind of bloat. Although Phoenix 1.3 is better in this regard than its predecessors, it still tempts developers to include their application code alongside the Phoenix code, either directly or as a so-called umbrella app.

Part of this temptation is the directory structure itself. Why is there a separate web subdirectory under gallows if we aren’t supposed to put non-web code somewhere?

Part of the temptation is also the fact that the ecto database interface code is included by default in every new Phoenix app. A pure web interface has no need of a database—all persistence work should be performed in services.

I strongly suggest you fight the temptation to stick everything in the Phoenix app. As we’ve been building our hangman game, we’ve been working hard to build code into separate, well defined applications. We’ll continue to do this as we add Phoenix integration.

If, in some future app, you fall victim to the pressure to put all your code into a single Phoenix basket, then it isn’t a disaster. But, when you finish, look at the result and ask yourself if it might be better structured as a set of isolated components.

6 Likes
2

I think in most cases it’s a good starting point. And that’s it. There is no big philosophy behind it. I was inspecting the way umbrella projects work recently because we were investigating how to deploy stuff separately to different nodes and still maintain one repository. The umbrella project is actually getting into the way in such set up.

But for most apps, simpler cases, it provides a good abstraction on the project structure, and convenient way to start the whole system in development, compile, run tests and other mix tasks.

It’s a starting point that can be later on migrated to more custom project structure. We don’t have to always agree with @pragdave, although he’s usually right. I like to think we can think on our own :wink:

2 Likes
3

@hubertlepicki Can you elaborate a bit on how umbrella project is making it harder to deploy your applications separately?

In my experience there’s no much difference between keeping the applications as an umbrella or as completely different projects. If you keep respect the boundaries between the applications, I think the only difference in that umbrella project lets you run mix tasks for everything at once which is really convenient for running tests.

One problem we had with umbrellas is that because they are compiled into one directory, they allow you to call functions from other applications without explicitly declaring dependencies in the mix.exs file. This is a big problem because you may even create a circular dependencies if you aren’t careful. To prevent this, we decided to build and test every application separately. Fortunately, we can do this on CI.

I guess the key is to keep the applications separated and decoupled. The way you do it - may it be an umbrella project or a set of independent applications in separate repositories - is not much of a difference.

3 Likes
4

Funny enough, it’s not the deployment at all. Deployment is fine, it’s super flexible with distillery and ansible set up. Again, my favourite abstraction of OTP application comes in as a savoir, I can decide where to push and start what stuff easily.

It’s development that I’m concerned with. I’d like to start a local cluster, that somehow resembles the production cluster, maybe with less nodes but similar topology. And while in production I don’t have all code loading onto all nodes, I don’t want to load all code onto all my nodes in development. So that I avoid bugs where I rely on something being loaded / started while on production it won’t even be available.

This set up requires custom scripting for local development, and you lose some nice helpers from umbrella project, but in turn you get the flexibility.

1 Like
5

I’m just a rank beginner, but came across this article that might be of interest? https://medium.com/matic-insurance/designing-scalable-application-with-elixir-from-umbrella-project-to-distributed-system-42f28c7e62f1

1 Like
6

I’m also interested in this topic. I’ve gone down the umbrella route and I think it’s great but I see it’s limitations. I’m curious how you would coordinate the applications using the standalone method. Seems like you’d need some sort of central registry you pass to each application.

I’d love to see an example repo using the standalone app approach.

7

There is a good discussion related to this topic here:

Personally I am a big fan of Dave’s approach - it just ‘makes sense’ to me.

I think Dave’s point is that if you go with an Umbrella app, it is still coupled, whereas his approach of totally independent apps, are, totally independent.

I’ve not heard him call it this, but I like to refer to it (or class it as) The Replaceable Component Architecture.

I can’t wait for his next course, which I hope ties in Ecto, deployment and goes more into sharing of or interacting with various data/components etc i.e. something more typical of a bigger real life app.

3 Likes
8

I agree I think Dave’s approach makes sense. I always separate my web and data layers using an umbrella app.

I’m more curious about the actual implementation. Umbrellas provide the ability to include apps as dependencies using the in_umbrella flag. I’m assuming with the standalone approach you’d have to do this yourself. I’m just not sure how you’d hook everything up.

1 Like
9

Dave Thomas use a path dependency instead.

1 Like
10

Let’s say you have two applications (A and B) in the web layer. You have a third application ( C ) in the data layer. If A and B both need to depend on C wouldn’t that start two instances of C? I would think it would make more sense to pass in a single instance of C rather than depend on it in that scenario. Seems like some sort of integration layer would be necessary. Am I thinking about this correctly?

1 Like
11

The problem remains the same wether You choose in_umbrella, or path. You might also have cross dependencies.

If You follow Dave Thomas’s advice, that would be no ecto, no context…

I do use ecto and umbrella. But I wanted to mention how he does the integration, in his course, with the help of the path option.

12

So you run into the same issues as the scenario above using an umbrella app? You’d have a separate instance of application C running for A and B?

13

AFAIK in_umbrella: true is a shorthand for path: "../<dep_name>", so I wouldn’t consider it a big difference.

14

Biggest difference lies in how the umbrella apps are generated, and in fact their configs. So in umbrella project all the relevant config files from all the apps will be loaded, i.e. config is shared by default.

When you build the project from multiple independent apps, you can either do that, or not. For example, I can have separate config/test.exs for each app, so I can test those in isolation (and even in parallel), mock some stuff in ui app and hit the database in db app (or other way around, as I please).

So the biggest advantage of not having umbrella project is that you don’t share the same config across all apps and it’s up to you how you configure it all.

5 Likes
15

Yup, config handling is different because of how it is generated. It’s nothing inherent, though, if you look at the config files (and the config_path: "../../config" setting in mix.exs), you’ll see you can easily change how it works to have a per-app config.

8 Likes