Workspace - A set of tools for working with Elixir monorepos

:tada: We are open sourcing workspace a set of tools for working with elixir monorepos, inspired by rust workspaces and nx.dev. Twitter announcement

You can find the docs here.

There are many reasons for having a monorepo but working with them may be cumbersome. Workspace provides a set of tools for working with big monorepos efficiently.

What is a workspace?

A workspace is nothing more than a git repository hosting multiple mix projects in any arbitrary folder structure. Path dependencies are used for cross project dependencies similarly to umbrellas.

The central concept of the workspace is the workspace graph. This is a directed acyclic graph where the nodes represent projects and the edges the dependencies between projects.

Pasted image 20240513130226

Workspace uses git to get the changes between two revisions. Knowing which files have changed we can deduce which project is modified or affected:

status

Running tasks

You can run any command from the root on all or a subset of the workspace projects. You can also run time consuming tasks only on the affected projects significantly improving CI times for large codebases.

run

Checking your workspace

When your workspace grows, it can go out of hand quickly. You need a way to check that your projects are properly defined. With the workspace.check task you can among other:

  • Ensure that specific dependencies are set on all projects, e.g. ex_doc.
  • Ensure that external dependencies versions match the expected ones.
  • Verify that no forbidden dependencies are defined.
  • Ensure that all projects have common build paths

Enforcing boundaries

One of the main purposes of monorepos is to help you split your codebase into independent, cohesive and reusable packages. You can enforce a clean architecture by tagging your projects and enforcing boundaries between them.

Creating a workspace

You can create a new workspace with the workspace.new scaffolder:

mix archive.install hex workspace_new

mix workspace.new your_workspace

Pasted image 20240513133314

Helper packages

The workspace repo is itself a workspace and comes with two extra small packages.

  • cli_options - provides an opinionated way to parse CLI options based on a schema similarly to NimbleOptions
  • cascade - is a small library for generating code for templates.

Disclaimer

Despite being stable (we successfully use it internally to manage a massive elixir codebase consisting of hundreds of packages) this is still a 0.x.x project and breaking changes may happen.

19 Likes

This is a really exciting start!

We’ve recently moved to GitHub - casey/just: 🤖 Just a command runner after we gave up on running various tasks through Mix. We will definitely give this a try and report back

3 Likes

Very interesting! How do you deal with IDE integration? Something specific? It does not work with ElixirLS on a hello-world example with package_a and package_b.

[Info  - 21:16:12] Updating incremental PLT
[Warn  - 21:16:13]     error: module PackageB is not loaded and could not be found

I don’t have any specific settings. Do path dependencies work in general for you?

Yes, they do. But I have to open each package as single folder in VSC for ElixirLS to work properly. So I thought maybe you have some tips how to open the full workspace in an IDE and still have it working. But that is probably a deep rabbit hole.

Also, a separate _build folder is maintained per package folder, right?

Would be curious about a small workspace example project with CI configs, some boundary checks and suggested folder layout for a bigger project. If that is not too much effort.

Looks really solid, I like that it provides enough structure and flexibility! Thanks for open-sourcing it.

1 Like

Yes, they do. But I have to open each package as single folder in VSC for ElixirLS to work properly. So I thought maybe you have some tips how to open the full workspace in an IDE and still have it working. But that is probably a deep rabbit hole.

Indeed, this is something that can be worked on.

Also, a separate _build folder is maintained per package folder, right?

This is up to you. We use common deps and _build paths, and also enforce it using workspace checks

1 Like

Would be curious about a small workspace example project with CI configs, some boundary checks and suggested folder layout for a bigger project. If that is not too much effort.

Workspace is dogfooding workspace so it is a simple example. It’s github CI is a nice example. You will see that in PRs only the affected projects are tested, while on the main branch the tasks are ran across all projects.

Regarding the project structure it’s up to you and how you wish to organize your code. In our case we prefer a domain oriented shallow folder structure. I will release the workspace demo project I used for the screenshots, which also includes some boundary checks.

Awesome! I didn’t notice, that workspace itself uses workspace, that’s cool! Yep, I guess I will have some fun time study-ing the codebase.

Thanks a lot!

2 Likes

Very cool. I’ll definitely be giving this ago after switching to a monorepo recently.

Are you using vscode’s workspaces? In the main menu there is “add folder to workspace”. I’ve done this for each folder and have an extension installed that opens a terminal for each one.

Yeah, vscode workspaces could be a solution. I was hoping for something a bit simpler, but with monorepos there is no such thing. :slight_smile: Maybe it’s good enough though.

Do it once, commit the json config to your source control and you’re done. Not sure how much simpler we can get than native IDE support? You could write a script to generate the workspace config file if you want.

1 Like

Wow, something like this-- the dependency checks, and boundary enforcement-- could help my organization’s goal in wrangling the various project repos, to get back to one monorepo.

Question: How well does this work with umbrella apps? Any special considerations or notes?

ie. most project repos are individual apps, but there exists one umbrella app. We were thinking of migrating all other projects under the umbrella.

Question: How well does this work with umbrella apps? Any special considerations or notes?

ie. most project repos are individual apps, but there exists one umbrella app. We were thinking of migrating all other projects under the umbrella.

It will not work with umbrellas since it is an umbrella alternative. You can move all umbrella apps out of the umbrella in any folder you wish, and it will work.

Hey @cmo, I’m trying to use workspaces, and ElixirLS is only picking up the top-level folder. Packages in nested folder are not recognized. Do you have any extra configuration to make it work?

.code-workspace - file:

{
	"folders": [
		{
			"path": ".",
			"name": "root"
		},
		{
			"path": "packages/package_a"
		}
	],
}

Modules in package_a are not loaded in ElixirLS.

It seems to work, when I include 2 subfolders. I get 2 ElixirLS processes, one for each elixir folder. But including the top-folder also into workspaces seems to confuse ElixirLS

Yes, exclude the top level folder. Mine is just:

{
	"folders": [
		{
			"name": "a",
			"path": "a"
		},
		{
			"name": "b",
			"path": "b"
		},
	],
	"settings": {
		"workbench.colorCustomizations": {}
	}
}

@pnezis you might want to explain the requirement to quote task args, e.g. mix workspace.run -t "deps.update --all" and the intricacies and pros/cons of using a common deps folder in the docs.

@cmo Although quoting works I would suggest to pass the task arguments after the return separator --. This way you will avoid escaping quotes. Thanks for bringing this up, I will update the docs accordingly.

mix workspace.run -t deps.update -- --all

Regarding the common deps path I have not included this in purpose, I don’t want to enforce any conventions. But you are right that a mention in the docs about the various options regarding the build artifacts folders would help. I will try to find an appropriate wording and will add it.

I did try that but

mix workspace.run -t deps.update -- --all                                                                                                             ─╯
==> nimble_options
Compiling 3 files (.ex)
Generated nimble_options app
==> cli_options
Compiling 5 files (.ex)
Generated cli_options app
==> workspace
Compiling 42 files (.ex)
Generated workspace app
** (CliOptions.ParseError) invalid option "all"
    (cli_options 0.1.0) lib/cli_options.ex:410: CliOptions.parse!/2
    (workspace 0.1.0) lib/mix/tasks/workspace.run.ex:225: Mix.Tasks.Workspace.Run.run/1
    (mix 1.16.2) lib/mix/task.ex:478: anonymous fn/3 in Mix.Task.run_task/5
    (mix 1.16.2) lib/mix/cli.ex:96: Mix.CLI.run_task/2
    c:/Users/simon/scoop/apps/elixir/current/bin/mix:2: (file)

edit: it’s a powershell thing I guess. Works in cmd :man_shrugging: