Migrating off of an umbrella app structure

How would one go about migration off an umbrella app structure in a way that wound’t require renaming apps folder into lib?

For context, we have a quite bit app that historically is using umbrella structure, e.g.:

apps/
  app1/
    lib/
      app1/
        # tree of app1 modules
      test/
        # ...
      priv/
        # ...
      mix.exs
  app2/
    lib/
      app2/
        # tree of app2 modules
      test/
        # ...
      priv/
        # ...
      mix.exs
  # several more

One approach for migrating off this would be to:

  • rename apps to lib,
  • merge all mix.exs into a single top-level one,
  • merge all priv folder contents into a single top-level one,
  • etc.

The reason I’m on the lookout for a different way is that doing steps above would cause a lot of move / rename git operations, which could make viewing git blame or navigating history for a change - potentially quite a bit harder for our developers.

Hence the question is: is there a way I could tell Elixir compiler that the .ex files containing app modules are located in non-standard location, e.g. apps instead of lib?

(I guess there absolutely have to be a single mix.exs as well as a single priv folder after all - I am perfectly OK to consolidate just these at the expense of having a git mv in their history as long as I don’t have to rename the apps folder itself)

1 Like

While you can set custom paths for compiled files I’d consider having a conventional folder structure to interact with on a daily basis is worth more than potentially needing to use multiple steps with blame in the cases you need to do that. Moving files really shouldn’t be a problem no matter the reason for it.

I’d probably refactor this in two steps: Leave the umbrella and move everything into one umbrella app. Once there’s only one app left (and everything works fine within the context of that one app) collapse the umbrella by moving the nested app to the root, updating mix.exs and retaining mix.lock.

4 Likes

We eventually migrated off the umbrella app, sticking to the officially recommended app structure, e.g. by having lib and test folders at the project root.

The whole process took under 2 weeks of work for a single person. During this time, I had to move around 3500 files. Probably 90% of this migration was automated, done via a bash script that moved the files around and did many replacements in the code using a combination of rg and sed commands.

rg / sed were largely used to find and replace occurrences of various forms of Application.get_env(), such as the following, for example:

Application.get_env(:my_app_1, :my_var)

:my_app_1 |> Application.get_env(:my_var)

:my_app_1
|> Application.get_env(:my_var)

But since it couldn’t catch all things, manual work was still required. And having a lot of tests helped too.

The most problematic part of migration, I think, was moving things related to the priv folder. I initially decided the new priv folder will have a structure like this:

priv/
  app_1/
    # stuff that previously lived in apps/app1/priv
  app_2/
    # stuff that previously lived in apps/app2/priv
  app_3/
    # stuff that previously lived in apps/app3/priv

Things mostly worked after this, but when they didn’t - it required a really hard manual searching of what got broken and why. Somehow this was much harder than when dealing with Application.get_env().

One of the bigger downsides of this migration is that git history is now polluted with a handful of “tectonic shift” commits, so searching how to make an original change in code using git blame became harder (although not impossible, of course). The script I mentioned above was also making commits after significant changes, so we ended up with over 150+ commits for various groups of changes in the code, to make it at least a little bit easier to review.

Having the migration scripted also allowed this work to be done without interrupting or “freezing” the main branch: people could still commit their code to the main branch, and I could rebase my branch against main & every time re-run the script “from scratch”. My manual changes were extracted into patch files, so they could be rolled on top of changes done automatically every time too (although setting this up turned out a bit problematic and didn’t work smoothly).

Our team is quite large & works on many things at once. Due to this fact, after umbrella app removal was merged, folks needed to rebase all of their open PRs against this mammoth change. Surprisingly, rebasing went quite smoothly: nowadays, git ships with good defaults that notice file movements & know how to apply changes that were made to file back when it was in its original location.

When git wasn’t able to figure out what to do, we used a few options:

  • find-renames, more on it is here, and
  • merge.directoryRenames to make certain kinds of file movements automatically,

So, git rebase became “tuned” like this:

git -c merge.directoryRenames=true rebase staging --strategy-option=find-renames=70

But again, this tuning wasn’t needed in 99% of cases.

Overall, everyone is quite happy with this transition. I could only wish we’d done it much earlier.

5 Likes

May I know the reason for migrating off umbrella apps?

Interesting writetup. Did you do this one app at a time, or was this en masse?

The reason we’ve started to seriously consider migrating off umbrella app, is a particular limitation we’ve noticed when using mix xref graph tool. It was a bit of a journey, so let me explain.

Recently several developers in the company reported that our application code compilation times have increased to the point they’ve become quite uncomfortable for day-to-day work.

We’ve started closely looking into what’s going on, and were able to identify and fix many low-hanging fruits using mix xref graph tool, which is a static analysis tool that reveals module dependencies. Someone suggested a gist with a tutorial for using mix xref on a real project. That gist was instrumental in helping me build a mental model around what kind of work needs to be done to reduce app re-compilation times.

After reaping the low-hanging fruit came a moment where we hit a limitation: mix xref graph will not show cross-application compile dependencies. It’s best illustrated based on example.

One of our apps is called api another one is called domain. mix xref graph --label compile-connected successfully shows dependencies between modules within a single app, but not between api and domain. While obviously there are many instances, where for example a controller from api app depends on a module from domain app.

Removing umbrella app helped us solve this, so running mix xref graph --label compile-connected now reveals the “full picture” and helped us see more opportunities for re-compilation optimization.

However, after years of running an umbrella app we’ve accumulated more issues with it:

  • the way we handled sharing code between apps - by extracting common code into yet another app - proved problematic,
  • the need to have a dedicated files like test_helper.ex, factory.ex etc. in each individual app,
  • a somewhat non-obvious connection between root project mix.exs respective file in each individual app,
  • calls to Path.join("../../..") across tests when in need to access something shared between the apps,
  • overall speed of test suite (suites for each individual app needs to run sequentially one after another),
  • mix test test/app/path/to/file_test.exs were never actual real path to files, but rather “path pattern” to look for a test file within the individual app,
  • some packages don’t work well with umbrellas (example)
  • etc.

So, small DX stuff like that. Not to mention we’ve developed a habit to attribute sporadic elixir-lsp misbehavings to our app being umbrella.

In the end of the day I tend to think no one on the team really understood or valued the benefits of umbrella app structure well enough to make a case for keeping it. Our app is pretty straightforward, REST web server application, and most of us craved for less obstacles in a day-to-day work.

2 Likes

We did it in an “all apps in one go” fashion.

The strategy was like this:

  1. move apps/my_app1/lib, apps/my_app2/lib, …, etc. folders to lib,
  2. move apps/my_app1/test, apps/my_app2/test, …, etc. folders to test,
  3. move apps/my_app1/priv, apps/my_app2/lib, …, etc. folders to priv,
  4. run & fix all tests.
  5. update files inside config to reference to my_app instead of my_app1, my_app2, …,
  6. search & replace all references to my_app1, my_app2, …, config values in lib and test folders,
  7. run & fix all tests.

Once I saw the number & nature of issues during step 4, I was more than ever positive that migrating off umbrella app is quite feasible. I think items 1-4 took two days of work, and the rest took another 7 days or work or so.

In the end it was also important to build & run a production release, and try to using it on a test data. For example, we’ve uncovered many previously not-covered-by-our-own-tests issues by running an external e2e suite.

2 Likes

Ah makes sense. If you were to go back would migrating each app as a single vertical of steps 1-7 one at a time reduce the step 4 iterations?

Potentially yes, I think so. But I am having a hard time wrapping my head around how I would approach doing it one-by-one, optimally.

My thinking process would follow original suggestion by @LostKobrakai above:

  • move apps/app1 to apps/app (basically just rename an app), this way - the goal would be to still have separate top-folders for each app’s code:
    • move apps/app1/lib to apps/app/lib/app,
    • move apps/app1/test to apps/app/test/app,
    • move apps/app1/priv to apps/app/priv/app,
  • run tests, see what failed & fix it,
  • go through priv / config too in the for this code too,
  • run & fix tests again,
  • start the app, see if it works (migrations are applied on app start correctly, translations are found in right place, etc.),
  • deploy it.

Repeat the steps for app2. Once all works, move the files once again so they end up lib, test, and priv folders.

I think I may be biased at the moment, or simply don’t see a more optimal strategy - but doing it like this appears more complicated to me. But again the process of migration is heavily context-dependent, so depending on the circumstances surrounding your app - this can be the best option for way forward out there.

1 Like