The issue was “resolved” by not duplicating config files. From Elixir v1.9, new apps have a single config/ directory in the umbrella, all other apps point to said config/ directory. So if you move to the same structure, then the problem disappears.
Unfortunately we cannot really track it otherwise, since the loading of config files is done dynamically (i.e. we don’t know upfront what we are going to load).
