What’s wrong with Umbrella Apps?

[BartOtten]

@sasajuric

Generally, my sentiment is that if the subapps are not deployed separately, the umbrella app doesn’t bring anything useful to the table, compared to single project + boundary

Amen.

@sasajuric

various paths that are printed by mix tasks (e.g. test IIRC) aren’t “clickable” in vscode (I couldn’t click to open the file in the editor), because the printed paths are missing the apps/myapp/ prefix

I wonder why you did not report a bug (or why it was closed when you did) as this simply seems like a bug to me.

[sasajuric]

@paulanthonywilson

I’m curious about how much faster.

I don’t have the numbers anymore, but the difference was visible. IIRC, the main reason was that various mix tasks (e.g. compile, test) processed apps sequentially. OTOH, when everything is in the same project, compilation of modules and tests can run concurrently on the entire codebase.

@BartOtten

I wonder why you did not report a bug (or why it was closed when you did) as this simply seems like a bug to me.

Whether that’s a bug is debatable, and I’m also unsure where should this be tackled (in the editor or in mix). Coupled with the fact that I personally don’t find umbrellas particularly useful, I didn’t spend any additional time on this. Instead I focused my effort on removing the umbrella, which brought various other benefits, as I mentioned earlier

[paulanthonywilson]

@sasajuric

@BartOtten

I wonder why you did not report a bug (or why it was closed when you did) as this simply seems like a bug to me.

Whether that’s a bug is debatable, and I’m also unsure where should this be tackled (in the editor or in mix)

I remember this definitely being an issue. I’ve just tried it out as I happened to be picking up an old project again, and the cmd-click now works from the root of the umbrella. ¯\(ツ)/¯ so I think it’s something that’s somehow got fixed in VSCode (or Elixir LS).

[cjbottaro]

Our xrefs sections became huge, so we had to ask why??

We used umbrella apps to determine what should be started as a part of Kubernetes deployments. But then I collapsed the umbrella app into a single app and used environment variables to determine what processes should be started. It was much more simple and we didn’t have all those dependencies (xref) problems.

Umbrellas can’t separate out your mix deps anyway… so they really just become about making separate releases. But disk space is cheap these days… why not have a single release and use env vars to determine what’s started?

[cjbottaro]

@sasajuric

Generally, my sentiment is that if the subapps are not deployed separately, the umbrella app doesn’t bring anything useful to the table, compared to single project + boundary

That was my original sentiment as well, but it’s very easy to conditionally start processes based on env vars (inspired by how Phoenix conditionally starts its endpoint).

Our monolithic Elixir app is responsible for over a dozen different Kubernetes deployments, all just using env vars to determine what processes are started.

[Kurisu]

@Kurisu

Another concern is for example I would like to have different priv folder per UI for gettext translations instead of using a single folder with domain translations.

After thinking more about this I realise that I could use multiple gettext backend modules in the same app and define different locations for their priv folder.

# Look for translations in my_app/admin_ui/priv/gettext instead of
# my_app/priv/gettext
use Gettext, otp_app: :my_app, priv: "admin_ui/priv/gettext"

I knew it is possible to define the location for the translations files but I didn’t think I can just define multiple modules such as MyAppWebGettext, MyAppAdminUiGettext etc.

I guess I could think of a way to have also different locations for assets.

And after reading all the posts in this thread, I guess I can be happier without my umbrella setup. Let’s embrace simplicity. ^^

Edit:

@Kurisu

For example I don’t know how I can tell mix phx.gen.html to put generated controller-view-templates files sometimes in admin_ui folder and the other times in the default_ui folder.

For generators such as phx.gen.html I would maybe create a sample project with the good namespace and generate the files inside that project, then I would copy and paste them in the desired folders of the project that I customized the folder structure.

Definitely I will give a try to thoses ideas in my next project to see how easier the things would become.

[sasajuric]

Yes, I also prefer this over umbrellas, and it’s the approach I advised to the clients in the past in the same scenario (microservices on k8s).

[LostKobrakai]

@cjbottaro

That was my original sentiment as well, but it’s very easy to conditionally start processes based on env vars (inspired by how Phoenix conditionally starts its endpoint).

Our monolithic Elixir app is responsible for over a dozen different Kubernetes deployments, all just using env vars to determine what processes are started.

There’s downsides to that as well though, as (only) runtime conditional starting of things still means you ship all the code everywhere. That might be a tradeoff, which can be made or might not. But which applications are put into releases can be customized in plain mix apps just as much as that can be done for umbrellas.

[dmitriid]

I think umbrella projects are not popular because very few people understand what they are, how to use them and why use them.

On top of that, there multiple issues mentioned here and elsewhere about troubles with tools, dependencies, testing etc.

Official docs just tell you to create an umbrella project without telling you any of the reasons why. (Well, some description it’s tacked to the end of the previous section, “Internal dependencies”).

So, back to your question:

What issues have people had that has put them off Umbrella Projects?

Most people never even started with umbrella projects because they are an unknown.

[paulanthonywilson]

Hi all,

Thanks again for this discussion. I thought it might be worth writing up my conclusions (which may not be the same as your conclusions).

Leaving the umbrella behind

[danschultzer]

Fascinating. Great writeup!

I also like umbrella projects, and have used it for nearly all production projects. My working environment is startups with very rapid prototyping. I haven’t had issues with keeping the projects DRY and the projects have always felt fast to build and deploy.

This is one example of an umbrella structure based on real world umbrella projects I’ve helped build:

• Ecto app
  • This is the meaty main app containing all business logic
• Phoenix app for API
  • Handling the API communication over HTTP/WebSockets and manages sessions
• Phoenix app for landing pages
• Phoenix app for internal admin
• Plug app acting as reverse proxy
  • Infrastructure required traffic had to be served on a single port in the docker container
  • Also handles requests to non-elixir web apps running in the container

The reason I always went with umbrella was that it seemed to give me the best balance between monolith and microservices. It helped me abstracting interfaces and concerns in a way that felt very natural and easy to understand.

I’ve actually went the other way of what @sasajuric described. I used umbrella structure to help clean up a messy Phoenix app. It helped remove odd code paths, clean up dependencies, and assets.

Also very recently showcasing the umbrella structure helped me convince a microservice heavy (AWS lambda) team to give Elixir a try and to keep it monolithic for faster development

The comments here are thought provoking. Thanks for initiating this discussion @paulanthonywilson! I’ll have to try some different approaches as well.

[cjbottaro]

A better solution is to just be unafraid to make top level namespaces in the main app (like how Phoenix creates MyApp and MyAppWeb)

That was a game changer for me.

Also realizing that my team doesn’t want true microservices. We simply want individually scalable deployments, but on a shared codebase.

[christhekeele]

@Kurisu

Multiple endpoints listening on different port with their own router, assets, gettext translations etc.

Yep, this is like the ideal use for them in my opinion. Good for code/config re-use between strongly related projects that have more than one deployable build target, and different resource/scaling needs.

@sasajuric

Generally, my sentiment is that if the subapps are not deployed separately, the umbrella app doesn’t bring anything useful to the table, compared to single project + boundary.

Precisely this!

@LostKobrakai

@cjbottaro

That was my original sentiment as well, but it’s very easy to conditionally start processes based on env vars (inspired by how Phoenix conditionally starts its endpoint).

Our monolithic Elixir app is responsible for over a dozen different Kubernetes deployments, all just using env vars to determine what processes are started.

There’s downsides to that as well though, as (only) runtime conditional starting of things still means you ship all the code everywhere.

There’s the further issue that you also have to conditionally configure other dependencies, like Ecto pool sizes, depending on the target environment. This is something you could get clever with using Mix.target for, but it still permutes your config access at every point it is used.

[sevenseacat]

I was trying to remember why I used to get frustrated working on one app I had, that was an umbrella app.

More directories can be a pain to navigate, especially if you are using a navigation tree. Having to be in different directories perform different mix tasks can also be a bit of a headache.

This was why. I was working on was a web application, that had one app for the Ecto/context layer and one for the Phoenix/GraphQL API - and it drove me nuts needing to constantly swap between folders to run different mix tasks.

[danschultzer]

it drove me nuts needing to constantly swap between folders to run different mix tasks

Unless it’s a one-off, I’m always using mix aliases at root with mix cmd to target apps:

[
  "my_task": "cmd --app app_1 mix my_task"
]

[Kurisu]

@danschultzer

Unless it’s a one-off, I’m always using mix aliases at root with mix cmd to target apps:

Interresting. Another solution is to have multiple tabs opened in the terminal and just have each of them pointed to the folder of a different app of the umbrella project.

I do similar setup for my code editor. Multiple workspaces for the same umbrella project. Depending on which app I want to work on, I just switch between the workspaces.

[christhekeele]

IIRC, mix commands run at the toplevel in umbrella apps “federate” into all subapps. So calling mix test will invoke it on all of them that have it defined. I’m not sure if it recognizes aliases within subapps, though—if not, you’d want to duplicate the aliases like described, and I think that honoring aliases would make a great feature request

[danschultzer]

IIRC, mix commands run at the toplevel in umbrella apps “federate” into all subapps.

Unfortunately that’s not how it works. The mix task in the dependency have to understand umbrella structure like how ExUnit handles it for that to work. So it depends on the dependency.

The default generated umbrella phoenix app will have a setup alias in each app and one at root level calling it recursively for each app using the cmd task.

[bit4bit]

i share that umbrella apps can be tricky for maintenance and operation
but, what about umbrella apps from perspective of tests?

example:
api → core → store
rpc → core

cd api; mix test → verify from api to store and not know rpc
cd core; mix test → verify from core to store

From perspective of test we run “isolate” from other apps , so this allows us to partially verify the hierarchy.

At least, for my team and i, allows to easy locate and test behaviors of the system.

[dgigafox]

Hi @al2o3cr do you have a public repo with this architecture? Mind if I take a look? I am working on a project with a similar take on separate compiled assets per UI