mix workspace.run -t deps.update --% --all
kudos to the powershell developers for making the behaviour different to the other shells 
With the common deps folder, I was thinking something that outlines what you need to be aware of/do if you choose to go down that path, e.g. the checks in workspace.exs to ensure everything is the same version.
This could solve a lot of problems for us. A lot of our code ended up in one large project, and smaller projects tend to bitrot.
One thing I’d like to see is if we could set up validation that one message or datatype emitted by one project is compatible with another project to prevent breakages between projects that communicate with each other. Sort of a test that spans multiple projects.
That may ne beyond the scope of this project.
One thing I’d like to see is if we could set up validation that one message or datatype emitted by one project is compatible with another project to prevent breakages between projects that communicate with each other. Sort of a test that spans multiple projects.
This should be handled by your tests. If you have configured your CI properly then when a project changes, all projects depending on it will also be tested. So if a breaking change is introduced and the proper tests are in place then the CI suite will break.
@mindreframer This is a sample workspace project with some boundaries rules enabled 
(all projects are mix hello world projects)
Awesome, this looks like a great example! Also great to see a working folder structure with apps / packages folders. Yeah, I think this is an absolutely needed alternative to umbrellas vs huge monolithic Phoenix apps.
Would it be possible to know how on what principles you structure your app with hundreds of packages?
Also how do you manage the workload between team members / teams, so that friction and code conflicts are reduced?
I dont see very often a large well structured Elixir app in the wild, hence my questions
Yeah, I think this is an absolutely needed alternative to umbrellas vs huge monolithic Phoenix apps.
Our main app is still a huge monolithic Phoenix app, the only difference is that instead of having all code under the app itself, we have moved independent pieces of code (e.g. third party api wrappers, utilities, broadway producers, etc.) to reusable packages that we add to the app as path dependencies. The main reason for this was the CI execution time and developer ergonomics. When for example you work on a broadway pipeline there is no need for the CI to run all API integration tests on your PRs since they are independent.
Using a poly-repo solution does not work for big enterprise systems. Managing dependencies across repos is a nightmare and there is no guarantee that a change on a package will not break something on apps/packages that depend on it.
We have managed to drastically reduce CI execution time by moving to a workspace. Also, given you have a well tested codebase, every change on every package will trigger all needed CI steps to all packages depending on it, ensuring that no breaking change is introduced. Of course some discipline is needed, e.g. when you bump a dependency you need to bump it on all projects using it, but this can be enforced with workspace checks.
As an advice I would say that you should start with a monolithic phoenix app or an umbrella and consider moving to a mono-repo/workspace only if the CI starts becoming a bottleneck.
Would it be possible to know how on what principles you structure your app with hundreds of packages?
The structure is very similar to the demo app. The only difference is that under packages we group packages based on the domain / scope:
packages
├── domain_1
│ ├── package_a
│ └── package_b
├── domain_2
│ └── package_c
└── shared
├── shared_package_1
└── shared_package_2
You can use though any folder structure that works for you. Since everything is a mix project, changing the structure to acomodate your needs is trivial, you only need to mv packages to the new folder and update the path dependencies (which can be automated).
Also how do you manage the workload between team members / teams, so that friction and code conflicts are reduced?
Code conflicts/friction is exactly the same as it would be if it was an umbrella app, a huge monolith or a polyrepo. If two developers need to modify the same file they would modify it no matter where it is located. This should happen rarely though if the codebase is decoupled.
What we do for helping with the daily operations, is to require every package to have at least one valid (team member) maintainer (this is something enforced by workspace checks). This works like CODEOWNERS on a package level, so we know who should review every change. Also this helps on handovers.
I dont see very often a large well structured Elixir app in the wild, hence my questions
Your questions were perceptive and straight to the point
Wow, that was quick! Thanks for the elaborate response!
That is exactly why I’m asking and also am very excited about this package! In umbrellas or normal phoenix apps the CI execution (compilation + tests) time grows in linear fashion and becomes enbearable rather quickly. Usual solution is test parallelisation, and it works OK, though it’s wasteful to run all the tests all the time, even though the change was maybe a readme adjustment. It’s just crazy.
By having a proper DAG between internal packages + nested hierarchy (as alternative to the flat folder structure in umbrellas) + tags and scopes, one can properly decide which tests should be executed and save CI execution time drastically
Also having domain-related packages grouped in a single folder is a great way to communicate intent and reduce the cognitive load of understanding how all the things are related.
I quite like this 2 level nesting. A flat packages folder is still prone to unbounded growth, keeping them in domain folders makes it so much nicer.
That was exactly my suspicion! Otherwise it would be too chaotic. Nice to have it confirmed.
Thanks a lot, I feel you have invested a ton of time in making a flexible, yet very structured and not over-engineered solution available for the Elixir community! Looking forward to some opensource projects adopting it.
Have a great day,
Roman
You will still need parallelisation for big projects, this time on the package level. workspace.run supports partitioned runs similarly to mix test. This has also the benefit that you can partition all time consuming CI steps, not only tests. If needed you can also have partitioned tests for big packages like now.
one can properly decide which tests should be executed
Totally agree, want to re-iterate that it is not only for tests and that the applicable packages are automatically picked by the workspace.run flags. For example:
# given that you use a common dependencies folder you only need to fetch and
# cache the external dependencies in your CI only from the root projects
mix workspace.run -t deps.get --only-roots
# format checks needs to run only on modified projects
mix workspace.run -t format --modified -- --check-formatted
# tests need to run to all projects affected by the changes
mix workspace.run -t test --affected
Thanks a lot for the feedback!
I’ve created an issue for vscode-elixir-ls and sent a PR to address this problem exactly because I caught it when decided to try Workspace. Hopefully, it will be approved, and then you just need to set the elixirLS.useCurrentRootFolderAsProjectDir flag and voila - it won’t try to go up to find the outermost mix.exs.
@pnezis thanks a lot for the project, it looks really promising and interesting!
May I ask, how to you deal with configs for each of these parts and for the bigger chunks up to the root apps as well? To be able to test independently, for example?
Do you have a central config for everything (like in umbrella) or keep them separately?