I like Umbrella projects and pretty much always use them for personal Elixir stuff, especially Nerves things.
But I don’t think this is a popular choice. I’ve never really understood why they are disliked. The two arguments I’ve read against them don’t check out for me: you can’t control the order of config loading (yes you can) and they are for deploying things separately (no they’re not).
Admittedly I have never worked on a huge Umbrella project; The massive monoliths I’ve worked on have all had flat architectures.
What issues have people had that has put them off Umbrella Projects?
I’ll let more experienced folks chime in regarding the technical reasons.
My personal experience from working in a few large umbrella projects (note that this is not due to the tech itself, just what happened with the folks using it) have mostly shown issues with premature and incorrect separation of concerns. This is a human problem, not a tech one. Symptoms of these are having sub-apps called core, core_db, utils… that are required everywhere in a project. Basically, instead of thinking in contexts within a non-umbrella app, folks try (incorrectly) to abstract those into apps. Sometimes, they mean well too.
If you can’t run tests in your sub-app without requiring other apps, you should likely have a flat structure and use contexts instead. If you have issues with dependencies (your own), you should reconsider the structure as well. Folks can use behaviours and define boundaries within the contexts if they are inclined (and get Mox for free out of the box if done properly). If you share the same database and have multiple sub-apps calling into it (and having issues with it), maybe your abstraction is incorrect. It is self imposed complexity most of the time. Some folks just want to do it too, because, they can.
These have been the bad experiences with it. Again, this is not related to the umbrella structure itself, but they did happen with folks that used it because they could.
I think by and large the problem is similar to “don’t use GenServers for code organization”. Likewise, often times people use umbrellas to organize their code, when umbrellas are best used when you have deployment heterogeneity (I think).
They are bad for organization because:
they create an extra layer of indirection in your file paths (apps/ directory)
it’s easy to wind up with intra-app dependencies that you lose track of.
grokking the mix.exs paths is tricky
some libraries don’t support umbrellas
A better solution is to just be unafraid to make top level namespaces in the main app (like how Phoenix creates MyApp and MyAppWeb). For example, I often have a Type module that contains project-specific Typing, and a Tools module, and a Db module.
I’ve found an umbrella structure really useful when you have multiple independent UIs to a single source-of-truth, especially when each UI wants its own asset compilation settings.
Some of the pain-points are mostly historical (for instance, the generator doesn’t even createconfig directories in sub-apps anymore, because the load order doesn’t work like that anyways) but there are definitely libraries that still rely on global config despite recommendations against it.
Enforcing this is actually one of the benefits of an umbrella project. Any subset of the apps in the umbrella could be deployed together in a release. If they can appear in the same release, then they need to depend on the same version of an external app.
Totally agree on this. That’s what I’m using umbrella for.
I’m still trying to figure out how to achieve that without umbrella but so far umbrella provides all I need out of the box for that with its generators :
Multiple endpoints listening on different port with their own router, assets, gettext translations etc. If I try to put those Phoenix apps in a single app I would have to do a lot more tweaks to achieve the “same goal”.
On the other hand some generators don’t support umbrella yet but I guess that could change in the future. For example mix phx.gen.release but it’s not a big deal.
Using mix xref ... to try and reduce compile-time dependencies between modules was near-impossible. Not sure if things improved since Elixir 1.11 on that but back then it was a huge turn-off for me.
A number of libraries that generate code inside your project don’t support umbrellas.
None of these are huge or actual show-stoppers but then again, working in a professional setting where you are paid to be productive introduces the problem of not having too much time to fiddle with such problems when they arise. So I eventually gave up, it didn’t seem to be worth the hassle to use umbrellas unless you really needed them – and several posters have provided legitimate needs for umbrellas already with which I fully agree.
The compile-time dependency thing is much improved in either 1.11 or 1.12 - it’s no longer possible to have “accidental” dependencies on other apps in the umbrella (where AppA references AppB.Something but does not declare AppB in deps) without a compiler warning. As a corollary, circular dependencies between subapps will always either trigger that warning (if the dep is not declared) or fail outright (if both subapps declare a dependency on each other).
I’d upvote this more than once if I could. mix xref doesn’t play nice at all with umbrella applications. It’s a PITA to make it work and find links between applications, and answering questions like “how much of this codebase is affected if I refactor this one function?” are very difficult to answer when your tools don’t understand umbrellas well enough.
This is a big point for me, not to mention that OTP applications are a runtime and deployment concern, not a code separation tool, so it doesn’t even make sense to use an umbrella for code organization in the first place. Tools like GitHub - sasa1977/boundary: Manage and restrain cross-module dependencies in Elixir projects offer a better way to tackle the “I don’t want code from this module to be called by this other module” problem in a saner way that doesn’t break most of the tools out there.
Also the fact that you have to append “in an umbrella app” whenever you ask someone for advice on fixing your weird bug should be a sign that something is off from the start.
I’d like for people to ask the question the other way around: why introduce umbrellas in the first place? rather than thinking that an umbrella is a good idea because reasons and then trying to find pain points later on.
Strange, I’d have guessed that sharing assets is more likely than having separate assets for different domains (I’m currently doing this, three domains at three different endpoints in the same VM, all sharing assets).
I run on 1.11, and that doesn’t solve the same problem, application boundaries are not the same as domain boundaries, and I believe that when most people reach for umbrellas, they want to solve the latter, not the former. There’s no reason to use umbrellas when just defining top level namespaces do the same, and don’t break tools like mix xref.
At least it’s not a problem the language itself is introducing, cross compatibility is a known problem in most software even nowadays.
There are also some other issues I’ve ran into, like having to force-compile the whole project because the order in which the sub applications were compiled somehow broke the whole umbrella, but I’d have to dig more into that bug again to provide a reproducible example(it had to do with running the project sometimes from root, sometimes from a sub app). I know for sure that wouldn’t have happened if the project wasn’t an umbrella, because it was precisely the “this module isn’t available because it’s in a different app” error.
That’s cool but doesn’t address the problem that running mix xref at the root of the umbrella basically doesn’t help you with anything that you would want when you normally run that command.
To me umbrellas are kind-of sort-of like the central Docker Compose / Kubernetes config file for a project that spans several microservices.
From what I’ve seen while working in Elixir companies, umbrellas are often liberally utilized as a stylistic / personal choice (usually by the CTO’s preferences) and are rarely used as they are technically meant to, leaving some of us to deal with the aftermath. Which is, again, utilizing mix xref to understand dependencies in a project better.
Having mix xref not work well in umbrellas leaves me with one less tool to enforce the single responsibility principle. Happily boundaries came along. It and mix xref – outside of umbrellas – have measurably improved the quality of my work (IMO).
Not saying everyone should be like me – but I personally will go out of my way to make a dynamic language a tad more predictable and easy to reason about upfront. Umbrellas stand directly in the way of that goal and that’s why I avoid them.
I have this non-umbrella app with two Phoenix endpoints, “api” and “web”: lib · trunk · Mikko Ahlroth / minisome · GitLab – And it really wasn’t any hassle at all. They’re both completely separate, yet in the same project. Now only one of them has frontend stuff, so setting up esbuild/webpack for two endpoints would have to be a custom job, but the Elixir side was easy to have separately.
A few months ago I started consulting on an umbrella project (about 10 apps). The first thing I did was deumbrellization of the project, which brought some benefits:
about 2k LOC less, due to removal of the repetitive boilerplate across subprojects
faster test and build times
simplified folder structure
better tooling support for simple projects
As an example of the last point, various paths that are printed by mix tasks (e.g. test IIRC) aren’t “clickable” in vscode (I couldn’t click to open the file in the editor), because the printed paths are missing the apps/myapp/ prefix. I found this extremely annoying and disruptive. This alone is for me the reason to avoid umbrella as much as possible.
Generally, my sentiment is that if the subapps are not deployed separately, the umbrella app doesn’t bring anything useful to the table, compared to single project + boundary. Even if boundary is not used, I find the benefits of umbrella don’t justify the issues mentioned above.
Thank you for sharing your settings. I will probably steal some ideas from it for my future projects. ^^
My concern is that if I use customized folder structure within a Phoenix app, I couldn’t take full adavantage of Phoenix generator anymore. The generators I think expect a given structure.
For example I don’t know how I can tell mix phx.gen.html to put generated controller-view-templates files sometimes in admin_ui folder and the other times in the default_ui folder.
Another concern is for example I would like to have different priv folder per UI for gettext translations instead of using a single folder with domain translations. Consider that my UIs don’t support the same set of locales and have their own domains of translation as well.
If there are a number of generators that don’t support umbrella project, I also feel like many can work out of the box without even being aware that the project is an umbrella. After all an umbrella app is just an app like another one in the first place.
I think it is a mater of preference but I’m really happy with umbrella as long as I keep the number of apps small and of course make sure all my apps depend only on the core app.
Thanks everyone for the discussion and sharing experiences. As I say, all the large projects I’ve worked on have been flat and have only experienced umbrellas on small (mostly personal) projects.
Much food for thought. I think the most significant issue with umbrellas reported here is around tooling, which is also an issue that I’ve also encountered. (There’s also that example and tutorials are written for flat structures and need to be tweaked for umbrellas). I really get how that could be a deal breaker.
I’m also inspired to take another look at boundary, which looks pretty useful. (I think the last time I looked at it things were in a very early state and I didn’t get round to investigating it further).